SharePoint 2013 Search Weirdness – Part 1: What's a Duplicate? (or where's my file?)

 

This is part of the "Search Weirdness" series…

Can't find my purchase order!

I created several purchase orders and uploaded them to a library. I can see them in the library, and when I use the library's search box I still see all of them (seven).

 

Now I go to the search box at the top of the page and try to find the purchase orders:

 

But I can only find two of them! I find the same two if I go to the Enterprise Search Center site.

 

Although there were more than two files in the search results, only the above two of the seven purchase orders were found.

 

Duplicates!

Turns out Search thinks six of these files are duplicates. They are near duplicates… The first six Purchase Orders vary only by Purchase Order number and Order Date. Not an uncommon thing when I what to order the same thing several months in a row. What's special about the seventh? It has a note added that says "Please ship via truck".

 

How do you find duplicates?

There are two options to deal with duplicates that both require an edit to the Search Results web part.

Option 1: Give the user a link to request the duplicates.

Option 2: Just show all of the duplicates.

 

Option 1: Give the user a link to request the duplicates

To find duplicates you need to edit the Search Results web part. The only problem with that is that by default the search box at the top of site pages points to _layouts/15/osssearchresults.aspx, which is not customizable by site owners or at all in Office 365. But we can customize the search results pages in an Enterprise Search Center.

1. Go to your search results page (you will have at least five, but will at least want to edit the results.aspx page), edit the page, and edit the Search Results web part.
2. Check the Show View Duplicates Link box.
   
3. Click OK and save the page.
4. Add an update to your "Site customization documentation". (You do have this don't you? )

Now your users can click the "VIEW DUPLICATES" link, if they can find it!

 

View the Duplicates

1. Perform the search…
2. Ask yourself "where's my purchase order?" 
3. Mouse over each of the purchase orders in the search results.
4. In the hover panel (pop out) click VIEW DUPLICATES.
5. Still can't find your PO? Click the back button and mouse over the next file and repeat!

 

 

More Weirdness?

I had seven purchase orders. When I moused over PO 12351 and clicked VIEW DUPLICATES I found four purchase orders: 12346, 12347, 12348 and 12351. When I moused over 12350 and clicked VIEW DUPLICATES I found six purchase orders: 12345, 12346, 12345, 12348, 12349 and 12350.

 

Option 2: Just show all of the duplicates

1. Go to your search results page (you will have at least five, but will at least want to edit the results.aspx page), edit the page, and edit the Search Results web part.
2. Click the Change Query button.
3. Click the Settings tab.
4. Click "Don't remove duplicates".
    
5. Click OK, OK and then save the page.
6. Add an update to your "Site customization documentation". (You do have this don't you? )

Goodie! I can now find all of my purchase orders! All seven of them.

 

 

All Done?

You have updated your Enterprise Search Center. What about all of those site search boxes?

 

Option 1: Train all of your users to use the library's search box or the Enterprise Search Center whenever they can't find what they are looking for with the site search box.

Option 2: Visit every site and, manually or using PowerShell, update the Search Navigation to use the Enterprise Search Center for all searches. (another reason you need a Search Administrator!)

 

Bug or Feature?

You decide…

You may want to take a look at these:

http://blogs.technet.com/b/fesiro/archive/2013/11/11/sharepoint-2013-search-near-duplicates-and-documentsignature.aspx

http://blogs.perficient.com/microsoft/2013/04/sharepoint-2013-search-not-display-all-results/

 

 

Now to pick another weirdness for the next article…

.

SharePoint Online: Sorry, something went wrong…There is not enough space on the disk.

 

My public Office 365 subscription shares resources with many other subscriptions. Somebody must be uploading a lot of stuff today!

And I was not even uploading anything! Just navigating around the tenant administration pages. Hopefully in a few minutes the SharePoint Online automation will move me, or someone else, to another server…

In the mean time, refreshing the page usually works, and sometimes I just have to wait a few minutes.I am a tenant after all. I don't own the building.

.

.

SharePoint Cincy - Let the SharePoint Search Genie Out of the Bottle!

 

Search Genie?

OK, that's a bit of a corny title, and a bit of a deception. My SharePoint Cincy presentation is really about the powerful things you can do with SharePoint 2013's search feature set… if you only had a Search Administrator.

If you've been to one of my governance classes or presentations then you have heard me lecture, practically preach, on why you need a Search Administrator. It's an easy job, only an hour or two a week, but it's also one of the most important jobs in the ongoing operation of SharePoint. Attend my presentation and meet the Genie!

SharePoint Cincy is this Friday! There's still time to register!

 

Oh, and there's another 21 sessions you might want to attend, and a lot of networking to do. See you Friday!

 

Let the SharePoint Search Genie Out of the Bottle!

Mike Smith

MAX Technical Training - MVP

What’s the single most important thing you can do to make SharePoint 2013, on premises or in the cloud, work for your users? Make stuff easy to find! Some say that SharePoint search is easy. Just install it and let it do its thing, and in Office 365 you don’t even have to install it. But… your users can’t find anything, or they find thousands of unwanted things. They don’t know where to look, they can’t spell your CEO’s name, they can’t find their own expense report and they can’t even find the lunch menu! There’s a powerful genie hiding in search who can find what your users are searching for. All it needs is an Administrator to turn it loose! So let’s let the search genie out of the bottle and discover the power of SharePoint 2013 Search Administration.

.

Run SharePoint 2013 Search Reports from PowerShell–Part 2!

 

First… see the original article here: http://techtrainingnotes.blogspot.com/2015/04/run-sharepoint-2013-search-reports-from.html

What if you want reports for each site collection? All we need is another function, a "Get-SPSIte –Limit ALL" and some code to generate unique file names and we can now create hundreds to thousands of Excel files!!! (Please make sure you never fill your server's drives with this!)

 

# This is the path to write the reports to:
$path = "c:\SearchReports\"


function Get-SPSearchReports ($farmurl, $searchreport, $path)
{
  # Report names and IDs
  $Number_of_Queries          = "21be5dff-c853-4259-ab01-ee8b2f6590c7"
  $Top_Queries_by_Day         = "56928342-6e3b-4382-a14d-3f5f4f8b6979"
  $Top_Queries_by_Month       = "a0a26a8c-bf99-48f4-a679-c283de58a0c4"
  $Abandoned_Queries_by_Day   = "e628cb24-27f3-4331-a683-669b5d9b37f0"
  $Abandoned_Queries_by_Month = "fbc9e2c1-49c9-44e7-8b6d-80d21c23f612"
  $No_Result_Queries_by_Day   = "5e97860f-0595-4a07-b6c2-222e784dc3a8"
  $No_Result_Queries_by_Month = "318556b1-cabc-4fad-bbd5-c1bf8ed97ab1"
  $Query_Rule_Usage_by_Day    = "22a16ae2-ded9-499d-934a-d2ddc00d406a"
  $Query_Rule_Usage_by_Month  = "f1d70093-6fa0-4701-909d-c0ed502e3df8"

  # create a unique filename for each site and report
  $sitename = $farmurl.Replace("http://","").Replace("https://","")
  $sitename = $sitename.substring(0,$sitename.IndexOf("/_layouts"))
  $sitename = $sitename.Replace("/","_").Replace(":","_").Replace(".","_")

  $filename = $path + $sitename + " " + 
              (Get-Variable $searchreport).Name + " " + 
              (Get-Date -Format "yyyy-mm-dd") + ".xlsx"
  #Write-Host "Creating $filename"

  $reportid = (Get-Variable $searchreport).Value

  $TTNcontent = "&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D" + $reportid

  # setup the WebRequest
  $webRequest = [System.Net.WebRequest]::Create($farmurl)
  $webRequest.UseDefaultCredentials = $true
  $webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*"
  $webRequest.ContentType = "application/x-www-form-urlencoded"
  $webRequest.Method = "POST"

  $encodedContent = [System.Text.Encoding]::UTF8.GetBytes($TTNcontent)
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()

  # get the data
  [System.Net.WebResponse] $resp = $webRequest.GetResponse();
    $rs = $resp.GetResponseStream();
    #[System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    #[byte[]]$results = $sr.ReadToEnd();
    [System.IO.BinaryReader] $sr = New-Object System.IO.BinaryReader -argumentList $rs;
    [byte[]]$results = $sr.ReadBytes(10000000);

  # write the file
  Set-Content $filename $results -enc byte
}


function Get-SPSearchReportsForSite ($url, $path)
{

  Get-SPSearchReports $url "Number_of_Queries" $path
  Get-SPSearchReports $url "Top_Queries_by_Day" $path
  Get-SPSearchReports $url "Top_Queries_by_Month" $path
  Get-SPSearchReports $url "Abandoned_Queries_by_Day" $path
  Get-SPSearchReports $url "Abandoned_Queries_by_Month" $path
  Get-SPSearchReports $url "No_Result_Queries_by_Day" $path
  Get-SPSearchReports $url "No_Result_Queries_by_Month" $path
  Get-SPSearchReports $url "Query_Rule_Usage_by_Day" $path
  Get-SPSearchReports $url "Query_Rule_Usage_by_Month" $path

}


Get-SPSite -Limit All | 
  foreach   {
    $url = $_.Url + "/_layouts/15/Reporting.aspx?Category=AnalyticsSiteCollection";
    #$path = (you might do a custom path for each set of reports)

    Write-Host "Getting files for $url"
    Get-SPSearchReportsForSite $url $path
  }

Run SharePoint 2013 Search Reports from PowerShell

Update: SharePoint 2016 version here: http://techtrainingnotes.blogspot.com/2018/02/run-sharepoint-2013-and-2016-search.html



Update! Need these reports for every site collection in the farm? See Part 2: http://techtrainingnotes.blogspot.com/2015/04/run-sharepoint-2013-search-reports-from_21.html

In my Search Administration class I stress that admins should dump the search reports on a regular basis as the data is only kept in detail for 14 days and in summary form for 35 months. But who wants to both run these reports at least once every 14 days, even they can remember to do so. So, PowerShell to the rescue… Schedule this script to run each weekend and your work is done.
The following script works for on premise SharePoint 2013. To work with Office 365 you will have to figure out how to include your credentials. The example included here works on premises by using "UseDefaultCredentials = $true".
After lots of hacking, detective work (see below) and just plain trial and error, here's the script:

# This is the URL from the Central Admin Search Service Usage Reports page:
$url = "http://yourCentralAdminURL/_layouts/15/reporting.aspx?Category=AnalyticsSearch&appid=ed39c68b%2D7276%2D46f7%2Db94a%2D4ae7125cf567"  

# This is the path to write the reports to (must exist):
$path = "c:\SearchReports\"




function Get-SPSearchReports ($farmurl, $searchreport, $path)
{
  # TechTrainingNotes.blogspot.com

  # Report names and IDs
  $Number_of_Queries          = "21be5dff-c853-4259-ab01-ee8b2f6590c7"
  $Top_Queries_by_Day         = "56928342-6e3b-4382-a14d-3f5f4f8b6979"
  $Top_Queries_by_Month       = "a0a26a8c-bf99-48f4-a679-c283de58a0c4"
  $Abandoned_Queries_by_Day   = "e628cb24-27f3-4331-a683-669b5d9b37f0"
  $Abandoned_Queries_by_Month = "fbc9e2c1-49c9-44e7-8b6d-80d21c23f612"
  $No_Result_Queries_by_Day   = "5e97860f-0595-4a07-b6c2-222e784dc3a8"
  $No_Result_Queries_by_Month = "318556b1-cabc-4fad-bbd5-c1bf8ed97ab1"
  $Query_Rule_Usage_by_Day    = "22a16ae2-ded9-499d-934a-d2ddc00d406a"
  $Query_Rule_Usage_by_Month  = "f1d70093-6fa0-4701-909d-c0ed502e3df8"


  $filename = $path + (Get-Variable $searchreport).Name + " " + (Get-Date -Format "yyyy-mm-dd")  + ".xlsx"
  $reportid = (Get-Variable $searchreport).Value

  $TTNcontent = "&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D" + $reportid

  # setup the WebRequest
  $webRequest = [System.Net.WebRequest]::Create($farmurl)
  $webRequest.UseDefaultCredentials = $true
  $webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*"
  $webRequest.ContentType = "application/x-www-form-urlencoded"
  $webRequest.Method = "POST"

  $encodedContent = [System.Text.Encoding]::UTF8.GetBytes($TTNcontent)
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()

  # get the data
  [System.Net.WebResponse] $resp = $webRequest.GetResponse();
    $rs = $resp.GetResponseStream();
    #[System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    #[byte[]]$results = $sr.ReadToEnd();
    [System.IO.BinaryReader] $sr = New-Object System.IO.BinaryReader -argumentList $rs;
    [byte[]]$results = $sr.ReadBytes(10000000);

  # write the file
  Set-Content $filename $results -enc byte
}



Get-SPSearchReports $url "Number_of_Queries" $path
Get-SPSearchReports $url "Top_Queries_by_Day" $path
Get-SPSearchReports $url "Top_Queries_by_Month" $path
Get-SPSearchReports $url "Abandoned_Queries_by_Day" $path
Get-SPSearchReports $url "Abandoned_Queries_by_Month" $path
Get-SPSearchReports $url "No_Result_Queries_by_Day" $path
Get-SPSearchReports $url "No_Result_Queries_by_Month" $path
Get-SPSearchReports $url "Query_Rule_Usage_by_Day" $path
Get-SPSearchReports $url "Query_Rule_Usage_by_Month" $path
 

The Detective Work…

I could not find anything documented on how the reports are called or details on things like the report GUIDs. So here's how I got there:

• Go the search reports page in Central Admin and press F12 to open the Internet Explorer F12 Developer Tools then:
  • Click the Network tab and click the play button to start recording.
  • Click one of the report links.
  • Double-click the link generated for the report in the F12 pane to open up the details.
  • Make note of the URL (It's the same as the report page!)
  • Note the Accept, and Content-Type Request Headers.
  • Click the Request Body tab.
  • Stare at 3000 characters in that string until your head really hurts, or until you recognize most of what is there is the normal page postback stuff like VIEWSTATE. So we need to find what's unique in the string. (It's the Report IDs.)
  • Click on each of the nine reports and copy out the report IDs.
  • With a lot of trial and error figure out what the minimum string needed is to generate the reports. (It's ""&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId" plus the report id.)
  • Find out how to do an HTTP POST using PowerShell. (Steal most of it from here: http://www.codeproject.com/Articles/846061/PowerShell-Http-Get-Post.)
  • Find some other needed .Net code and convert the C# to PowerShell.
  • Fill in some gaps with PowerShell putty …….

.

Setting a SharePoint Group's Description Using PowerShell

 

The SharePoint Group object has a property named "Description", but it appears to be ignored. When you edit a group in the browser there is no "Description" field, but there is one named "About Me" that is a rich text field. Turns out there's a few secrets you need to know to set this seemly simple little property.

Secret #1: Many of the group's properties are stored in another object, the SPWeb.SiteUserInfoList object.

Secret #2: The internal name of "About Me" is "Notes".

The PowerShell:

#create the group as usual and then retrieve it...
$web = $site.RootWeb;
$group = $web.SiteGroups["Test Group with HTML desc"];

#find the group's SiteUserInfoList info...
$groupInfo = $web.SiteUserInfoList.GetItemById($group.id);

#update the text...
$groupInfo["Notes"] = "<div>This is <strong>bold</strong> and this is <em>italics</em></div>";

#and save it...
$groupInfo.Update();

 

For a C# example see the answer in this discussion: http://stackoverflow.com/questions/968819/change-description-of-a-sharepoint-group

 

.

PowerShell: Finding all SharePoint Lists with Lookup Columns

 

Did you ever want to find all of the columns of a certain type? All Lookups, all Calculated or all Managed Metadata columns? All you have to do is look at the Fields object "TypeDisplayName" property. While the example script below is looking for Lookup columns, you could modify it to look for any kind of column.

 

Find all Lookup Columns

This will find all Lookup columns in the entire farm!

Notes:

• Many of the out of the box lists are "custom" lists and will have fields that look like user added columns. Exclude those with the $TTNExcludeLists variable.
• Field that are Hidden or FromBaseType are not typically user created and are excluded in this example.

$TTNExcludeLists = "Solution Gallery", 
                "Workflow Tasks", 
                "Master Page Gallery"

Get-SPSite -Limit All | Get-SPWeb -Limit All | 
  Select -ExpandProperty Lists | 
  Where { -Not ($TTNExcludeLists -Contains $_.Title) } | 
  Select -ExpandProperty Fields | 
  Where { $_.TypeDisplayName -eq "Lookup" -and 
          $_.Hidden -eq $false -and 
          $_.FromBaseType -eq $false } | 
  Select {$_.ParentList.ParentWebUrl}, 
         {$_.ParentList}, 
         Title

 

TypeDisplayName

If you would like to see the list of all of the column types used in your site or farm you can run a script like this:

Get-SPWeb "http://yourServer/sites/YourSite" |
  Select -ExpandProperty Lists | 
  Select -ExpandProperty Fields |
  Select TypeDisplayName –Unique |

  Sort

It may take a long time to run (WARNING!) but this will list all of the columns in the farm:

Get-SPSite -Limit All | Get-SPWeb -Limit All | 
  Select -ExpandProperty Lists | 
  Select -ExpandProperty Fields |
  Select TypeDisplayName -Unique|
  Sort TypeDisplayName

 

Here's the list I got from my test farm:

TypeDisplayName
---------------
All Day Event
Attachments
Audience Targeting
Calculated
Channel Alias
Check Double Booking
Choice
Computed
Content Type Id
Content Type ID
Counter
Cross Project Link
Date and Time
Event Type
File
Free/Busy
Guid
Hold Status
Hyperlink or Picture
Integer
Lookup
Managed Metadata
Moderation Status
Multiple lines of text
Number
Number of Likes
Number of Ratings
Out of Policy
Outcome choice
Page Separator
Permission Level
Person or Group
Publishing HTML
Publishing Image
Publishing Schedule End Date
Publishing Schedule Start Date
Rating (0-5)
Recurrence
Related Items
Resources
Single line of text
Summary Links
ThreadIndex
User Agent Substrings
Variations
Yes/No

 

.

Word Stemming in SharePoint 2013 Search

 

Word Stemming

Word stemming is when a search engine resolves a word to its root form and then returns variations of the word.

For example, search for one of these and return the rest:

• run, ran runner, running, runs
• fish, fishes, fisher, fishing

 

SharePoint 2013 Word Stemming

SharePoint 2013 only does "stemming" on singulars and plurals. So all we get for "run" is "runs" and for "fish" is "fishes". The exact stemming depends on the language.

SharePoint 2007 and 2010 did support an expanded form of word stemming, but you had to enable it in the search results web parts.

 

So what do we do?

Search using "OR".

   fish OR fishing OR fisher

   run OR ran OR running OR runs

Remember that the Boolean operators like "OR" must be typed in upper case and that mixtures of AND and OR usually will need parentheses.

   race AND (run OR ran OR running OR runs)

 

oh well…

 

.

SharePoint 2013 Items Removed with Search Result Removal Return from the Dead!

 

In SharePoint 2010, Removed was Removed!

In SharePoint 2010 there's a search feature called "Search Result Removal" that made an item immediately disappear from user's search results. All you had to do was enter the URL to the problem item and click Remove Now. SharePoint then remove the item from the index and wrote a Crawl Rule to make sure the content was ignored by the crawler in all future searches.

The 2010 request to remove:

 

The auto-created Crawl Rule:

SharePoint 2010 had a interesting bug here. Uppercase letters in the URL would cause the the removal request to be ignored!

 

In SharePoint 2013 and Office 365, Removed is Just a Temporary Thing! Maybe only a few seconds!

We don't have Crawl Rules in SharePoint 2013 and Office 365. When you request the removal of an item or a URL from the search index, it just deletes it from the index. But… on the next crawl it adds it back! That next crawl could be an hour away, or only seconds, depending on your crawl schedules and the luck of your timing.

Nowhere can I find any documentation that says this!

The screen for removal requests is similar to what we had with 2010, but with no mention about Crawl Rules.

 

Removal Straight from the Crawl Logs

Both versions let you browse the crawl logs, find a problem item, click the dropdown on the item and click Remove the Item From The Index. In the Crawl Logs you can remove one item at a time while the Removal pages will let you exclude and entire path.

In 2013 we are just told that the item will be removed. In 2010 we are also told that a crawl rule will be created.

 

References:

Remove URLs from search results (SharePoint Server 2010
https://technet.microsoft.com/en-us/library/ff621095(v=office.14).aspx

Delete items from the search index or from search results in SharePoint Server 2013
https://technet.microsoft.com/en-us/library/jj219587.aspx

SharePoint 2013 Search File Extensions

 

Just a simple list of file extensions that have some kind of default support in the SharePoint 2013 Search Service.

FH = SharePoint 2013 has a File Handler and can index the content of this file.
FT = Is included in the default on premises File Type List.
365 = Is included in the Office 365 / SharePoint Online File Type List. (as of 3/5/15)

Extension	FH	FT	365	Type	MimeType
.ascx		x	x	ASP.NET Control
.asm		x	x	ASP Web Service
.asp		x	x	ASP web page
.aspx		x	x	ASP.NET web page
.csv		x	x	Comma separated values
.cxx		x	x
.def		x	x
.doc	x	x	x	Microsoft Word	application/msword
.docm	x	x	x	Microsoft Word	application/vnd.ms-word.document.macroEnabled.12
.docx	x	x	x	Microsoft Word	application/vnd.openxmlformats-officedocument.wor...
.dot	x	x	x	Microsoft Word	application/msword
.dotm	x			Microsoft Word	application/vnd.ms-word.template.macroEnabledTemp...
.dotx	x	x	x	Microsoft Word	application/vnd.openxmlformats-officedocument.wor...
.eml	x	x	x	Email Message	message/rfc822
.gif	x	x	x	Graphics Interchange Format	image/gif
.html	x	x	x	Web Page	text/html
.infopathml	x			Microsoft InfoPath	text/xml
.jpg	x			JPEG	image/jpeg
.jsp		x	x	Java server page
.mht	x	x	x	Web Archive	multipart/related
.mhtml		x	x	Web page archive format
.msg	x	x	x	Outlook Item	application/vnd.ms-outlook
.nws		x	x
.obd	x			Microsoft Office Binder	application/vnd.ms-binder
.obt	x			Microsoft Office Binder	application/vnd.ms-binder
.odc		x	x	Office Data Connection File
.odp	x	x	x	OpenDocument Presentation	application/vnd.oasis.opendocument.presentation
.ods	x	x	x	OpenDocument Spreadsheet	application/vnd.oasis.opendocument.spreadsheet
.odt	x	x	x	OpenDocument Text	application/vnd.oasis.opendocument.text
.one	x		x	OneNote	application/msonenote
.pdf	x	x	x	PDF	application/pdf
.pot	x			Microsoft PowerPoint	application/vnd.ms-powerpoint
.potm	x			Microsoft PowerPoint	application/vnd.ms-powerpoint.template.macroEnabl...
.potx	x			Microsoft PowerPoint	application/vnd.openxmlformats-officedocument.pre...
.ppam	x			Microsoft PowerPoint	application/vnd.ms-powerpoint.addin.macroEnabled.12
.pps	x			Microsoft PowerPoint	application/vnd.ms-powerpoint
.ppsm	x			Microsoft PowerPoint	application/vnd.ms-powerpoint.slideshow.macroEnab...
.ppsx	x			Microsoft PowerPoint	application/vnd.openxmlformats-officedocument.pre...
.ppt	x	x	x	Microsoft PowerPoint	application/vnd.ms-powerpoint
.pptm	x	x	x	Microsoft PowerPoint	application/vnd.ms-powerpoint.presentation.macroE...
.pptx	x	x	x	Microsoft PowerPoint	application/vnd.openxmlformats-officedocument.pre...
.pub	x	x	x	Microsoft Publisher	application/x-mspublisher
.rtf	x			Rich Text	text/rtf
.txt	x	x	x	Text	text/plain
.vcf	x			vCard	text/x-vcard
.vcs	x			vCalendar	text/x-vCalendar
.vdw	x	x	x	Visio	application/vnd.visio
.vdx	x	x	x	Visio	application/vnd.visio
.vsd	x	x	x	Visio	application/vnd.visio
.vsdm	x	x	x	Visio	application/vnd.ms-visio.drawing.macroEnabled
.vsdx	x	x	x	Visio	application/vnd.ms-visio.drawing
.vss	x	x	x	Visio	application/vnd.visio
.vssm	x	x	x	Visio	application/vnd.ms-visio.stencil.macroEnabled
.vssx	x	x	x	Visio	application/vnd.ms-visio.stencil
.vst	x	x	x	Visio	application/vnd.visio
.vstm	x	x	x	Visio	application/vnd.ms-visio.template.macroEnabled
.vstx	x	x	x	Visio	application/vnd.ms-visio.template
.vsx	x	x	x	Visio	application/vnd.visio
.vtx	x	x	x	Visio	application/vnd.visio
.xlb	x		x	Microsoft Excel	application/vnd.ms-excel
.xlc	x		x	Microsoft Excel	application/vnd.ms-excel
.xls	x	x	x	Microsoft Excel	application/vnd.ms-excel
.xlsb	x	x	x	Microsoft Excel	application/vnd.ms-excel.sheet.binary.macroEnable...
.xlsm	x	x	x	Microsoft Excel	application/vnd.ms-excel.sheet.macroEnabled.12
.xlsx	x	x	x	Microsoft Excel	application/vnd.openxmlformats-officedocument.spr...
.xlt	x		x	Microsoft Excel	application/vnd.ms-excel
.xml	x	x	x	XML Document	text/xml
.xps	x			Microsoft XML Paper Specification	application/vnd.ms-xpsdocument
.zip	x	x	x	ZIP Archive	application/zip

 

Notes:

• SharePoint indexes all content in a site, even if not in the File Types list. You can search for any of the data in the columns of the library to find the file. For example you can search for "jpg" and find everything that has jpg in the filename, description, or other column. You can't search for the file using "filetype:jpg" though.
• If the file extension is added to the File Types list then SharePoint treats the item as a "file" and you can then search using "filetype:jpg".
• If the file extension is in the File Types list AND there is a Format Handler or iFilter then the content of the file will also be indexed.
• You can dump the list of installed Format Handlers using this PowerShell script:
• $ssa = Get-SPEnterpriseSearchServiceApplication
  Get-SPEnterpriseSearchFileFormat -SearchApplication $ssa | ft –AutoSize
• The above list of files has nothing to do with Blocked File Types (files users cannot upload)
  https://technet.microsoft.com/en-us/library/cc262496.aspx
• Default crawled file name extensions and parsed file types in SharePoint Server 2013
  https://technet.microsoft.com/en-us/library/jj219530.aspx
• HOW TO: Implement a custom iFilter in SharePoint 2013
  http://blogs.technet.com/b/sharepointdevelopersupport/archive/2013/05/13/how-to-implement-a-custom-ifilter-in-sharepoint-2013.aspx

SharePoint 2013 Search Spelling Suggestions Confusion?

 

Over the last three months I have been working on a Search Administration class that has a focus on improving the end user search experience. If you have attended one of my governance classes or consulting sessions then you have heard me preach on the need for a "Search Administrator". If you are interested in this area of administration then check out my new class: Microsoft SharePoint 2013 Search Administration. The next class is 3/11/15!

SharePoint 2013 search is not always well documented. As an example, read the description of Search Spelling Suggestions here: https://technet.microsoft.com/en-us/library/jj591607.aspx While very factual, it just does not tell me enough…

So off to the blogs…

First problem is that it seems a lot of people mix up Search Query Suggestions and Search Spelling Suggestions. They are not the same! Query Suggestions are offered while the user types in the search box. Spelling Suggestions are offered only in the search results pages, after the user executes the search.

A Spelling Suggestion:

(“corydoras” is kind of catfish. )

 

Second problem? It seems due to the lack of extensive documentation that a lot of people are guessing.

• One blog says that "only the first level of Terms is taken into account".
• One even said that spelling suggestions could be used to map between terms. (Must have been thinking about Query Suggestions.)
• Another says that while searching for a term found in the second level of a term set, and no items are found, that the parent level's term will be offered as the suggestion.
• Some call the Query Spelling Inclusions term set the "Static" dictionary.

All of those are wrong.

 

Here's what my trial and error has found:

• There are four sources for spelling suggestions:
  • The default static spelling dictionaries (static out of the box dictionaries).
  • The default dynamic spelling dictionary (dynamically generated from your content). This is also called a "content-aligned spelling dictionary".
  • The Query Spelling Inclusions term set (manually entered).
  • The Query Spelling Exclusions term set (manually entered).
• Spelling suggestions are based on the closest matches in the default spelling dictionaries and the Query Spelling Inclusions list. Only one suggestion will be displayed. It appears that a phonic / sound alike match is being used, so the properly spelled and misspelled words must be similar in length and pattern.
• You can’t edit the default static or dynamic spelling dictionaries.
• Static dictionaries
  • The static dictionary is a canned list of words. (Something like the Word dictionaries.)
  • The user can change the language dropdown and see words from another language.
• Dynamically created dictionaries
  • The dynamic dictionary is created by search as content is indexed. I.e. it's based on words commonly found in your content.
  • The user can change the language dropdown, but this does not change offered words (at least from my testing).
  • Support for dynamically created dictionaries depends on the language. See https://technet.microsoft.com/en-us/library/jj219499.aspx
  • For dynamic spelling correction to work, you should have at least several thousand medium-sized documents. The default settings require that a word occur in at least 1000 documents to be included in the dictionary.
  • The dynamic dictionary is updated once a night and can be a long running process. (The default timeout is 6 hours!)
• Query Spelling Inclusions / Exclusions
  • You must have a configured Managed Metadata Service.
  • The terms must be added to the auto-created Query Spelling Inclusions term set.
  • You manually enter a list of words into an include or exclude list of terms.
  • You can only include single words, not phrases in the term sets.
  • While sub-terms can be entered, they are treated no differently than top level terms. (Some documentation says they are ignored.) Some documentation says terms at the same sublevel can be used as “Do you mean”, but testing does not show this.
  • It may take ten minutes or more for updates to the term sets to show up in search results. (Search Custom Dictionaries Update timer job)

As there are a number of articles about adding words to the Query Spelling Inclusions term set using Managed Metadata Services, code and PowerShell, I will not cover that here.

 

What about the options available in the PowerShell commands?

There are two PowerShell cmdlets that can be used to manage the on premises SharePoint's spelling options. Again the TechNet article is factual, but not too clear on the Static vs. Dynamic topic.

    Get-SPEnterpriseSearchQuerySpellingCorrection
    Set-SPEnterpriseSearchQuerySpellingCorrection

Note: These cmdlets are currently only for on premises SharePoint. It looks like Office 365 is set to use the Dynamic dictionary.

You can see the spelling suggestion options using:

    $ssa = Get-SPEnterpriseSearchServiceApplication
    Get-SPEnterpriseSearchQuerySpellingCorrection -SearchApplication $ssa

   

SpellingDictionary = Static / Dynamic

Many of the blogs state that you can choose either Static or Dynamic as the SpellingDictionary value, and that by selecting one of these you would exclude your manually entered Query Spelling Inclusions . The confusion seems to be around the definitions of the words Static and Dynamic. My testing shows that:

  Dynamic = use the list of words found in your content, plus the Query Spelling Inclusions term set

  Static = use the static built-in / out of the box dictionary, plus the Query Spelling Inclusions term set

As an example, I do not have "SharePoint" or "SharePint" in my Query Spelling Inclusions term set, but I do have "corydoras". When I do a search for "SharePint" I get the following only when SpellingDictionary is set to Static: (this word is in the canned dictionary)

   

When searching for a term in the Query Spelling Inclusions term set like "corydoras" I get results regardless of if SpellingDictionary is set to Static or Dynamic.

   

Why don't I get any help when SpellingDictionary is set to Dynamic and I search for "SharePint"?

   

Take a look at the TermFrequencyThreshold property. Using the defaults SharePoint would need to find at least 1000 documents that contain the word "SharePoint". My testing sample set of documents is not quite that big. If I change it from 1000 to 20 and run the "Spelling dictionary update" timer job then I can start to get useful results from "Did you mean?" for "SharePint". (i.e. "SharePoint" was in at least 20 of my sample documents.)

   

 

 

Note: Using PowerShell to change from Static to Dynamic, or the reverse, immediately impacts user searches.

 

Additional Links of Interest

This article, Search in SharePoint 2013 knowledge articles for Systems Center Operations Manager, hints at more details about the search spelling suggestions feature.

The spelling related timer jobs are listed in this page: Timer job reference (SharePoint 2013)

.

SharePoint 2013 Query Suggestion Files

 

Over the last three months I have been working on a Search Administration class that has a focus on improving the end user search experience. If you have attended one of my governance classes or consulting sessions then you have heard me preach on the need for a "Search Administrator". If you are interested in this area of administration then check out my new class: Microsoft SharePoint 2013 Search Administration.

I have been experimenting with SharePoint 2013 search Query Suggestions and found that there are some interesting aspects of the import files that don't seem to be documented anywhere.

• One phrase per line.
• The “Always Suggest” file does not need quotes, but they are acceptable.
• The “Never Suggest” file must have quotes around multiword phrases.
• Files can have blank lines.
• Files with duplicates cannot be uploaded.
• The order of the phrases is not important. The phrases will be sorted in the dropdown. For that matter, if you import the file and then export it, the items are in alphabetical order.
• It's a good idea to keep your own backup of the files. Do not depend on the export as the order has been changed. You may also want to keep a "documented" version of the file with some comments. (Remember to remove the comments before uploading!)

I tried most of the ways to add comments, but none of them worked. You may want to keep two copies of your suggestion files, one with comments and one without.

Example of an Always Suggest file:

Example of a Never Suggest file:

In this example we are hiding common misspellings learned by search's automatic query suggestions. We are also dealing with the fact we don’t have a "Cincinnati region". It’s really the South Western Ohio Region.

 

Tip: If you are working with an on premises SharePoint 2013 then you could add a thesaurus entry to expand the user's search for "Cincinnati Region" to include "South Western Ohio Region". 

.

Working with Quick Launch from PowerShell

 

The following will work with both SharePoint 2010 and 2013. It will also work with SharePoint 2007 if you "manually" create the SPWeb object.

The basic steps to add a new link to Quick Launch:

1. Get the SPWeb object for your site.
2. Get the SPWeb.Navigation.QuickLaunch object.
3. Create a new node .
4. Add the new node to an existing node (like "Lists" or "Libraries") or the root of Quick Launch.
5. Done… No need to call Update() on anything!

Note: This will work with publishing sites as long as you are not using Managed Metadata Navigation.

 

Get your web object and the QuickLaunch object:

$web = Get-SPWeb http://yourServer/sites/yourSite
$quicklaunch = $web.Navigation.QuickLaunch

You can explore your existing Quick Launch from here. Just type $quicklaunch and press enter to see your headings / top level nodes. Type $quicklaunch | where {$_.title -eq "Lists"} to see the links in one of the headings. You could even list all of the headings and their children with this one: $quicklaunch | select -ExpandProperty Children | Select {$_.Parent.Title}, {$_.Title}.

 

Create a new node object:

Each item added to Quick Launch is an SPNavigationNode object. It has three parameters: the text to display, the URL to the linked resource and $true if the link is external to SharePoint or $false if the link is internal to SharePoint. It appears that the $true/$false in the third parameter is used to see if the URL is validated as a real SharePoint URL or not. I could still add SharePoint links by leaving it as $true.

Example for a SharePoint link: (note the $false)

$navnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode("Get help!", "/sites/helpsite", $false)

Example for Bing or link external to SharePoint:

$navnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode("Bing", http://www.bing.com, $true)

Example for JavaScript:

$navnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode("Say Hello!", "javascript:alert('hello')", $true)

Example for JavaScript to open a new dialog box:
(Note the "`" in front of the "$" is needed because "$" means something special in PowerShell. (a variable follows))

$navnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode("Add a new task", "JavaScript:var options=SP.UI.`$create_DialogOptions();options.url='/sites/Publishing/Lists/Tasks/NewForm.aspx?RootFolder=&IsDlg=1';options.height = 400;void(SP.UI.ModalDialog.showModalDialog(options))", $true)

For more on JavaScript in Quick Launch see SharePoint: JavaScript in Quick Launch and Top Link Bar! and SharePoint: Opening a 2010 Dialog Box from Quick Launch.

 

Add the node to Quick Launch

To add the new node as a "heading" just add it to the Quick Launch object:

$quicklaunch.AddAsFirst($navnode)             or .AddAsLast

To add the new node as a child of a "heading" then use Where to find that heading:

$heading = $quicklaunch | where {$_.title -eq "Lists"}
$heading.Children.AddAsLast($navnode)

To add the new node after an existing node (i.e. not as First or Last) you will need to retrieve the existing node and then use the .Add method.

$existingLink = $heading.Children | where {$_.title -eq "Search the web with Bing"}

$navnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode("Search the web with Google", "http://www.google.com", $true)

$heading.Children.Add($navnode,$existingLink)

 

Delete a link?

Sure… Just retrieve the node and then call Delete.

$heading | Select -ExpandProperty Children | where { $_.Title -eq "Tasks" } | ForEach { $_.Delete() }

If you know that there's exactly one node that matches the Where test then you can shorten it to this:

($heading | Select -ExpandProperty Children | where { $_.Title -eq "Goog
le" }).Delete()

 

Bulk updates?

Sure… with extreme caution!  Danger! Will Robinson! Danger! Do the following at your own risk. No liability assumed, batteries not included!

Here's an example to add a Help link to every site in a site collection or to all site collections in the entire farm. (You way want to filter out the My Sites and the publishing sites!)

First confirm what you are about to change!

Get-SPSite http://sharepoint/sites/training | Get-SPWeb -Limit All | Select URL, Title
   or
Get-SPSite -Limit All | Get-SPWeb -Limit All | Select URL, Title

Create the new link / node.

$navnode = New-Object Microsoft.SharePoint.Navigation.SPNavigationNode("HELP!", "http://yourServer/sites/yourHelpSite", $true)

Now add it to every site in a site collection!

Get-SPSite http://sharepoint/sites/training | Get-SPWeb -Limit All | ForEach { $_.Navigation.QuickLaunch.AddAsFirst($navnode) }

or for all site collections in the farm:

Get-SPSite -Limit All | Get-SPWeb -Limit All | ForEach { $_.Navigation.QuickLaunch.AddAsFirst($navnode) }

 

And if you want to delete all of those helpful links?

Get-SPSite http://sharepoint/sites/training | Get-SPWeb -Limit All | ForEach { ($_.Navigation.QuickLaunch | where {$_.Title -eq "Help!"}).Delete()  }

 

What about a Publishing Site?

Publishing Sites use the "Navigation" editor for links and it does not permit "URLs" that don't begin with "http". This means that you cannot add JavaScript to a Quick Launch link using that editor. For SharePoint 2010 you can use the workaround here, or for 2010 and 2013 you can use the PowerShell above. (PowerShell to the rescue!) For more on JavaScript in Quick Launch see SharePoint: JavaScript in Quick Launch and Top Link Bar! and SharePoint: Opening a 2010 Dialog Box from Quick Launch.

The PowerShell above will work as long as in your Navigation settings you have not selected "Managed Navigation" or "same as parent".

 

References:

MSDN's article showing C# examples: (This is for 2010 but applies to 2007-2013.)

https://msdn.microsoft.com/en-us/library/office/ms427791(v=office.14).aspx

SharePoint Search and Dates

 

I created a file on 2/7/2015. I searched SharePoint using "Write=2/7/2015" and could not find the file! When I found it by keywords, search said it was created on 2/7/2015.

   

The library said it was created on 2/7/2015.

   

When I searched using "Write=2/8/2015" I found it!

Turns out that search works in a different time zone! Here's the quote from this MSDN article:

"All date/time values must be specified according to the UTC (Coordinated Universal Time), also known as GMT (Greenwich Mean Time) time zone."

My testing is in Eastern Standard Time (EST), so GMT is currently 5 hours "in the future". When using GMT my file was created at 2/8/2015 12:50 AM.

 

Now try to explain that to all of your SharePoint users!

 

By the way… what time zone are you, or your users using? In my Office 365 subscription it's defaulting to PST when I create new site collections. I guess I should go check all of my other site collections and see I remembered to change to them all to EST when I created them!

 

.

Hide Documents from Delve

 

HideFromDelve!

Just stumbled on this buried in the middle of a Search Schema document.To hide documents from Delve you basically create a new Site Column named HideFromDelve as a Yes/No column, add it to your library and then set this column to Yes for everything you don't want to show up in Delve.

https://support.office.com/en-us/article/Manage-the-search-schema-in-SharePoint-Online-d4fab46d-ba41-4c03-9d4c-32b5b33198b6?ui=en-US&rs=en-US&ad=US#BKMK_HideFromDelveSteps

 

If you've not heard of Delve then start here: https://support.office.com/en-US/article/What-is-Office-Delve-1315665a-c6af-4409-a28d-49f8916878ca?ui=en-US&rs=en-US&ad=US

 

.

PowerShell Sorting Tip for Enumerations

 

Do you ever have a PowerShell script that just won't sort right?

The problem? PowerShell makes assumptions, and not always the ones you would make. For example let's get a list of SharePoint lists and sort them by their BaseTemplate property:

    $mtg = Get-SPWeb http://maxsp2013wfe/sites/Meetings
    $mtg.Lists | Sort BaseTemplate | Select BaseTemplate, Title

Is the result you would expect?

Not what I first expected… but there's a hint hiding there though… Why is the BaseTemplate column right aligned as if was numeric?

If we take the above script and pipe it to Get-Member we find that BaseTemplate is not a string! It's an object of some kind. It's an SPListTemplateType object!

So let's see what that is… Doing a Bing on Microsoft.SharePoint.SPListTemplateType reveals that it is an enumeration, which internally is a number. If you look at the list in the MSDN article you will see both the text and numeric values of the base types.

What we need is the display text for the enumeration, and we can get that by using the ToString() method. As that would then be an expression, we need to add the annoying curly brackets and the $_. notation.

    $mtg.Lists | Sort { $_.BaseTemplate.ToString() } | Select BaseTemplate, Title

And now we get:

And that's more like it! (except for the right align stuff that going on)

Convert the column in the Select to a string and all's well!

    $mtg.Lists | Sort {$_.BaseTemplate.ToString()} | Select {$_.BaseTemplate.ToString()}, Title

 

So what were the PowerShell assumptions?

1. To display the text value of the enumeration, and align it as a number.
2. To sort on the numeric value of the enumeration.

(And you know what "assume" does, right?) 

 

.

SharePoint Auditing and Site Content Administration using PowerShell

 

If you have attended one of my SharePoint Saturday PowerShell presentations or read my PowerShell series at SharePoint Pro magazine, then you know a bit about what's possible with PowerShell. This Friday. January, 30th, I will be delivering one of the more fun classes I do: "SharePoint 2010 and 2013 Auditing and Site Content Administration using PowerShell". If you are a SharePoint administrator and need to "find and inventory stuff" in SharePoint then you need this class! Whatever is hiding in SharePoint is visible to the PowerShell user, if you know where to look.

Class details are here: http://www.maxtrain.com/Classes/ClassInfo.aspx?Id=119394
You can join us at MAX Technical Training in Cincinnati or attend remotely from your office or home.

This class is available world wide though Microsoft training partners
and the Microsoft Courseware library. Just ask for course 55095.

 

MS-55095 SharePoint 2010 and 2013 Auditing and Site Content Administration using PowerShell

This one day instructor-led class is designed for SharePoint 2010 and 2013 server administrators and auditors who need to query just about anything in SharePoint. The class handout is effectively a take home cheat sheet with over 175 PowerShell scripts plus the general patterns to create your own scripts. These scripts cover:

• getting lists / inventories of servers, services web applications, sites, webs, lists, libraries, items, fields, content types, users and much more
• finding lists by template type, content type and types of content
• finding files by user, content type, file extension, checked out status, size and age
• finding inactive sites
• finding and changing SharePoint Designer settings and finding and resetting customized pages
• inventorying and managing features
• deleting and recycling files and list items
• inventorying users and user permissions and finding out “who can access what”
• creating sites, lists and libraries
• uploading and downloading files
• general tips for counting, reformatting and exporting results;
• drilling up and down the SharePoint object model
• and much more…

 

Prerequisites: You should have good SharePoint skills as and end user and administrator along with some practical experience with PowerShell.

 

.