SharePoint 2013 Search Weirdness – Part 1: What's a Duplicate? (or where's my file?)

 

This is part of the "Search Weirdness" series…

Can't find my purchase order!

I created several purchase orders and uploaded them to a library. I can see them in the library, and when I use the library's search box I still see all of them (seven).

 

Now I go to the search box at the top of the page and try to find the purchase orders:

 

But I can only find two of them! I find the same two if I go to the Enterprise Search Center site.

 

Although there were more than two files in the search results, only the above two of the seven purchase orders were found.

 

Duplicates!

Turns out Search thinks six of these files are duplicates. They are near duplicates… The first six Purchase Orders vary only by Purchase Order number and Order Date. Not an uncommon thing when I what to order the same thing several months in a row. What's special about the seventh? It has a note added that says "Please ship via truck".

 

How do you find duplicates?

There are two options to deal with duplicates that both require an edit to the Search Results web part.

Option 1: Give the user a link to request the duplicates.

Option 2: Just show all of the duplicates.

 

Option 1: Give the user a link to request the duplicates

To find duplicates you need to edit the Search Results web part. The only problem with that is that by default the search box at the top of site pages points to _layouts/15/osssearchresults.aspx, which is not customizable by site owners or at all in Office 365. But we can customize the search results pages in an Enterprise Search Center.

1. Go to your search results page (you will have at least five, but will at least want to edit the results.aspx page), edit the page, and edit the Search Results web part.
2. Check the Show View Duplicates Link box.
   
3. Click OK and save the page.
4. Add an update to your "Site customization documentation". (You do have this don't you? )

Now your users can click the "VIEW DUPLICATES" link, if they can find it!

 

View the Duplicates

1. Perform the search…
2. Ask yourself "where's my purchase order?" 
3. Mouse over each of the purchase orders in the search results.
4. In the hover panel (pop out) click VIEW DUPLICATES.
5. Still can't find your PO? Click the back button and mouse over the next file and repeat!

 

 

More Weirdness?

I had seven purchase orders. When I moused over PO 12351 and clicked VIEW DUPLICATES I found four purchase orders: 12346, 12347, 12348 and 12351. When I moused over 12350 and clicked VIEW DUPLICATES I found six purchase orders: 12345, 12346, 12345, 12348, 12349 and 12350.

 

Option 2: Just show all of the duplicates

1. Go to your search results page (you will have at least five, but will at least want to edit the results.aspx page), edit the page, and edit the Search Results web part.
2. Click the Change Query button.
3. Click the Settings tab.
4. Click "Don't remove duplicates".
    
5. Click OK, OK and then save the page.
6. Add an update to your "Site customization documentation". (You do have this don't you? )

Goodie! I can now find all of my purchase orders! All seven of them.

 

 

All Done?

You have updated your Enterprise Search Center. What about all of those site search boxes?

 

Option 1: Train all of your users to use the library's search box or the Enterprise Search Center whenever they can't find what they are looking for with the site search box.

Option 2: Visit every site and, manually or using PowerShell, update the Search Navigation to use the Enterprise Search Center for all searches. (another reason you need a Search Administrator!)

 

Bug or Feature?

You decide…

You may want to take a look at these:

http://blogs.technet.com/b/fesiro/archive/2013/11/11/sharepoint-2013-search-near-duplicates-and-documentsignature.aspx

http://blogs.perficient.com/microsoft/2013/04/sharepoint-2013-search-not-display-all-results/

 

 

Now to pick another weirdness for the next article…

.

SharePoint 2013 Search Weirdness

In my presentation at last week's SharePoint Cincy 2015 I had a slide titled "Search Weirdness" that raised a few eyebrows. It had a list of things that are, well, unexpected at the least. As I keep finding more of these "weird" things I thought I should put together a little series of blog articles on them.
Here they are:

1. Part 1: What's a Duplicate? (or where's my purchase order?)
2. Part 2: Can't Find My Videos!
3. Part 3: When do Daily and Weekly Search Alerts Get Sent?
4. Part 4: More Duplication Weirdness
5. Part 5: REST API Ignores Duplicates
6. Part 6:???

Also see:

• SharePoint Search Tip: Search for "title" where there is no "title"!
• SharePoint Search and Dates
• Word Stemming in SharePoint 2013 Search
• SharePoint 2013 Search Spelling Suggestions Confusion?
• SharePoint 2013 Search File Extensions
• Run SharePoint 2013 Search Reports from PowerShell

.

SharePoint Online: Sorry, something went wrong…There is not enough space on the disk.

 

My public Office 365 subscription shares resources with many other subscriptions. Somebody must be uploading a lot of stuff today!

And I was not even uploading anything! Just navigating around the tenant administration pages. Hopefully in a few minutes the SharePoint Online automation will move me, or someone else, to another server…

In the mean time, refreshing the page usually works, and sometimes I just have to wait a few minutes.I am a tenant after all. I don't own the building.

.

.

SharePoint Cincy - Let the SharePoint Search Genie Out of the Bottle!

 

Search Genie?

OK, that's a bit of a corny title, and a bit of a deception. My SharePoint Cincy presentation is really about the powerful things you can do with SharePoint 2013's search feature set… if you only had a Search Administrator.

If you've been to one of my governance classes or presentations then you have heard me lecture, practically preach, on why you need a Search Administrator. It's an easy job, only an hour or two a week, but it's also one of the most important jobs in the ongoing operation of SharePoint. Attend my presentation and meet the Genie!

SharePoint Cincy is this Friday! There's still time to register!

 

Oh, and there's another 21 sessions you might want to attend, and a lot of networking to do. See you Friday!

 

Let the SharePoint Search Genie Out of the Bottle!

Mike Smith

MAX Technical Training - MVP

What’s the single most important thing you can do to make SharePoint 2013, on premises or in the cloud, work for your users? Make stuff easy to find! Some say that SharePoint search is easy. Just install it and let it do its thing, and in Office 365 you don’t even have to install it. But… your users can’t find anything, or they find thousands of unwanted things. They don’t know where to look, they can’t spell your CEO’s name, they can’t find their own expense report and they can’t even find the lunch menu! There’s a powerful genie hiding in search who can find what your users are searching for. All it needs is an Administrator to turn it loose! So let’s let the search genie out of the bottle and discover the power of SharePoint 2013 Search Administration.

.

Run SharePoint 2013 Search Reports from PowerShell–Part 2!

 

First… see the original article here: http://techtrainingnotes.blogspot.com/2015/04/run-sharepoint-2013-search-reports-from.html

What if you want reports for each site collection? All we need is another function, a "Get-SPSIte –Limit ALL" and some code to generate unique file names and we can now create hundreds to thousands of Excel files!!! (Please make sure you never fill your server's drives with this!)

 

# This is the path to write the reports to:
$path = "c:\SearchReports\"


function Get-SPSearchReports ($farmurl, $searchreport, $path)
{
  # Report names and IDs
  $Number_of_Queries          = "21be5dff-c853-4259-ab01-ee8b2f6590c7"
  $Top_Queries_by_Day         = "56928342-6e3b-4382-a14d-3f5f4f8b6979"
  $Top_Queries_by_Month       = "a0a26a8c-bf99-48f4-a679-c283de58a0c4"
  $Abandoned_Queries_by_Day   = "e628cb24-27f3-4331-a683-669b5d9b37f0"
  $Abandoned_Queries_by_Month = "fbc9e2c1-49c9-44e7-8b6d-80d21c23f612"
  $No_Result_Queries_by_Day   = "5e97860f-0595-4a07-b6c2-222e784dc3a8"
  $No_Result_Queries_by_Month = "318556b1-cabc-4fad-bbd5-c1bf8ed97ab1"
  $Query_Rule_Usage_by_Day    = "22a16ae2-ded9-499d-934a-d2ddc00d406a"
  $Query_Rule_Usage_by_Month  = "f1d70093-6fa0-4701-909d-c0ed502e3df8"

  # create a unique filename for each site and report
  $sitename = $farmurl.Replace("http://","").Replace("https://","")
  $sitename = $sitename.substring(0,$sitename.IndexOf("/_layouts"))
  $sitename = $sitename.Replace("/","_").Replace(":","_").Replace(".","_")

  $filename = $path + $sitename + " " + 
              (Get-Variable $searchreport).Name + " " + 
              (Get-Date -Format "yyyy-mm-dd") + ".xlsx"
  #Write-Host "Creating $filename"

  $reportid = (Get-Variable $searchreport).Value

  $TTNcontent = "&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D" + $reportid

  # setup the WebRequest
  $webRequest = [System.Net.WebRequest]::Create($farmurl)
  $webRequest.UseDefaultCredentials = $true
  $webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*"
  $webRequest.ContentType = "application/x-www-form-urlencoded"
  $webRequest.Method = "POST"

  $encodedContent = [System.Text.Encoding]::UTF8.GetBytes($TTNcontent)
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()

  # get the data
  [System.Net.WebResponse] $resp = $webRequest.GetResponse();
    $rs = $resp.GetResponseStream();
    #[System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    #[byte[]]$results = $sr.ReadToEnd();
    [System.IO.BinaryReader] $sr = New-Object System.IO.BinaryReader -argumentList $rs;
    [byte[]]$results = $sr.ReadBytes(10000000);

  # write the file
  Set-Content $filename $results -enc byte
}


function Get-SPSearchReportsForSite ($url, $path)
{

  Get-SPSearchReports $url "Number_of_Queries" $path
  Get-SPSearchReports $url "Top_Queries_by_Day" $path
  Get-SPSearchReports $url "Top_Queries_by_Month" $path
  Get-SPSearchReports $url "Abandoned_Queries_by_Day" $path
  Get-SPSearchReports $url "Abandoned_Queries_by_Month" $path
  Get-SPSearchReports $url "No_Result_Queries_by_Day" $path
  Get-SPSearchReports $url "No_Result_Queries_by_Month" $path
  Get-SPSearchReports $url "Query_Rule_Usage_by_Day" $path
  Get-SPSearchReports $url "Query_Rule_Usage_by_Month" $path

}


Get-SPSite -Limit All | 
  foreach   {
    $url = $_.Url + "/_layouts/15/Reporting.aspx?Category=AnalyticsSiteCollection";
    #$path = (you might do a custom path for each set of reports)

    Write-Host "Getting files for $url"
    Get-SPSearchReportsForSite $url $path
  }

Run SharePoint 2013 Search Reports from PowerShell

Update: SharePoint 2016 version here: http://techtrainingnotes.blogspot.com/2018/02/run-sharepoint-2013-and-2016-search.html



Update! Need these reports for every site collection in the farm? See Part 2: http://techtrainingnotes.blogspot.com/2015/04/run-sharepoint-2013-search-reports-from_21.html

In my Search Administration class I stress that admins should dump the search reports on a regular basis as the data is only kept in detail for 14 days and in summary form for 35 months. But who wants to both run these reports at least once every 14 days, even they can remember to do so. So, PowerShell to the rescue… Schedule this script to run each weekend and your work is done.
The following script works for on premise SharePoint 2013. To work with Office 365 you will have to figure out how to include your credentials. The example included here works on premises by using "UseDefaultCredentials = $true".
After lots of hacking, detective work (see below) and just plain trial and error, here's the script:

# This is the URL from the Central Admin Search Service Usage Reports page:
$url = "http://yourCentralAdminURL/_layouts/15/reporting.aspx?Category=AnalyticsSearch&appid=ed39c68b%2D7276%2D46f7%2Db94a%2D4ae7125cf567"  

# This is the path to write the reports to (must exist):
$path = "c:\SearchReports\"




function Get-SPSearchReports ($farmurl, $searchreport, $path)
{
  # TechTrainingNotes.blogspot.com

  # Report names and IDs
  $Number_of_Queries          = "21be5dff-c853-4259-ab01-ee8b2f6590c7"
  $Top_Queries_by_Day         = "56928342-6e3b-4382-a14d-3f5f4f8b6979"
  $Top_Queries_by_Month       = "a0a26a8c-bf99-48f4-a679-c283de58a0c4"
  $Abandoned_Queries_by_Day   = "e628cb24-27f3-4331-a683-669b5d9b37f0"
  $Abandoned_Queries_by_Month = "fbc9e2c1-49c9-44e7-8b6d-80d21c23f612"
  $No_Result_Queries_by_Day   = "5e97860f-0595-4a07-b6c2-222e784dc3a8"
  $No_Result_Queries_by_Month = "318556b1-cabc-4fad-bbd5-c1bf8ed97ab1"
  $Query_Rule_Usage_by_Day    = "22a16ae2-ded9-499d-934a-d2ddc00d406a"
  $Query_Rule_Usage_by_Month  = "f1d70093-6fa0-4701-909d-c0ed502e3df8"


  $filename = $path + (Get-Variable $searchreport).Name + " " + (Get-Date -Format "yyyy-mm-dd")  + ".xlsx"
  $reportid = (Get-Variable $searchreport).Value

  $TTNcontent = "&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D" + $reportid

  # setup the WebRequest
  $webRequest = [System.Net.WebRequest]::Create($farmurl)
  $webRequest.UseDefaultCredentials = $true
  $webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*"
  $webRequest.ContentType = "application/x-www-form-urlencoded"
  $webRequest.Method = "POST"

  $encodedContent = [System.Text.Encoding]::UTF8.GetBytes($TTNcontent)
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()

  # get the data
  [System.Net.WebResponse] $resp = $webRequest.GetResponse();
    $rs = $resp.GetResponseStream();
    #[System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    #[byte[]]$results = $sr.ReadToEnd();
    [System.IO.BinaryReader] $sr = New-Object System.IO.BinaryReader -argumentList $rs;
    [byte[]]$results = $sr.ReadBytes(10000000);

  # write the file
  Set-Content $filename $results -enc byte
}



Get-SPSearchReports $url "Number_of_Queries" $path
Get-SPSearchReports $url "Top_Queries_by_Day" $path
Get-SPSearchReports $url "Top_Queries_by_Month" $path
Get-SPSearchReports $url "Abandoned_Queries_by_Day" $path
Get-SPSearchReports $url "Abandoned_Queries_by_Month" $path
Get-SPSearchReports $url "No_Result_Queries_by_Day" $path
Get-SPSearchReports $url "No_Result_Queries_by_Month" $path
Get-SPSearchReports $url "Query_Rule_Usage_by_Day" $path
Get-SPSearchReports $url "Query_Rule_Usage_by_Month" $path
 

The Detective Work…

I could not find anything documented on how the reports are called or details on things like the report GUIDs. So here's how I got there:

• Go the search reports page in Central Admin and press F12 to open the Internet Explorer F12 Developer Tools then:
  • Click the Network tab and click the play button to start recording.
  • Click one of the report links.
  • Double-click the link generated for the report in the F12 pane to open up the details.
  • Make note of the URL (It's the same as the report page!)
  • Note the Accept, and Content-Type Request Headers.
  • Click the Request Body tab.
  • Stare at 3000 characters in that string until your head really hurts, or until you recognize most of what is there is the normal page postback stuff like VIEWSTATE. So we need to find what's unique in the string. (It's the Report IDs.)
  • Click on each of the nine reports and copy out the report IDs.
  • With a lot of trial and error figure out what the minimum string needed is to generate the reports. (It's ""&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId" plus the report id.)
  • Find out how to do an HTTP POST using PowerShell. (Steal most of it from here: http://www.codeproject.com/Articles/846061/PowerShell-Http-Get-Post.)
  • Find some other needed .Net code and convert the C# to PowerShell.
  • Fill in some gaps with PowerShell putty …….

.

PowerShell: Collection was modified; enumeration operation may not execute

 

While the following example is for PowerShell, the same applies to C# code.

Collections that are being modified do not like to be processed with FOR EACH.
"Collection was modified; enumeration operation may not execute"

For example:

$web = Get-SPWeb someurl...

ForEach($list in $w.Lists)
{
  if (!$list.Hidden)
  {
    # do something to change the list
    # like add a new view...

    $list.Views.Add(.......)
    # error! 
    # "Collection was modified; enumeration operation may not execute..."
  }
 }
$web.Dispose()

 

The Fix

Change the FOR EACH to a FOR.

change:

  foreach($l in $web.Lists)
     {

to:

   for ($i=0; $i -lt $web.lists.count; $i++)
     {
        $l = $w.lists[$i]

 

And if you are deleting items in the collection…

Walk the collection in reverse order!

   for ($i=$web.lists.count-1; $i –gt 0; $i--)
     {
        $l = $web.lists[$i]

.

Setting a SharePoint Group's Description Using PowerShell

 

The SharePoint Group object has a property named "Description", but it appears to be ignored. When you edit a group in the browser there is no "Description" field, but there is one named "About Me" that is a rich text field. Turns out there's a few secrets you need to know to set this seemly simple little property.

Secret #1: Many of the group's properties are stored in another object, the SPWeb.SiteUserInfoList object.

Secret #2: The internal name of "About Me" is "Notes".

The PowerShell:

#create the group as usual and then retrieve it...
$web = $site.RootWeb;
$group = $web.SiteGroups["Test Group with HTML desc"];

#find the group's SiteUserInfoList info...
$groupInfo = $web.SiteUserInfoList.GetItemById($group.id);

#update the text...
$groupInfo["Notes"] = "<div>This is <strong>bold</strong> and this is <em>italics</em></div>";

#and save it...
$groupInfo.Update();

 

For a C# example see the answer in this discussion: http://stackoverflow.com/questions/968819/change-description-of-a-sharepoint-group

 

.

PowerShell: Finding all SharePoint Lists with Lookup Columns

 

Did you ever want to find all of the columns of a certain type? All Lookups, all Calculated or all Managed Metadata columns? All you have to do is look at the Fields object "TypeDisplayName" property. While the example script below is looking for Lookup columns, you could modify it to look for any kind of column.

 

Find all Lookup Columns

This will find all Lookup columns in the entire farm!

Notes:

• Many of the out of the box lists are "custom" lists and will have fields that look like user added columns. Exclude those with the $TTNExcludeLists variable.
• Field that are Hidden or FromBaseType are not typically user created and are excluded in this example.

$TTNExcludeLists = "Solution Gallery", 
                "Workflow Tasks", 
                "Master Page Gallery"

Get-SPSite -Limit All | Get-SPWeb -Limit All | 
  Select -ExpandProperty Lists | 
  Where { -Not ($TTNExcludeLists -Contains $_.Title) } | 
  Select -ExpandProperty Fields | 
  Where { $_.TypeDisplayName -eq "Lookup" -and 
          $_.Hidden -eq $false -and 
          $_.FromBaseType -eq $false } | 
  Select {$_.ParentList.ParentWebUrl}, 
         {$_.ParentList}, 
         Title

 

TypeDisplayName

If you would like to see the list of all of the column types used in your site or farm you can run a script like this:

Get-SPWeb "http://yourServer/sites/YourSite" |
  Select -ExpandProperty Lists | 
  Select -ExpandProperty Fields |
  Select TypeDisplayName –Unique |

  Sort

It may take a long time to run (WARNING!) but this will list all of the columns in the farm:

Get-SPSite -Limit All | Get-SPWeb -Limit All | 
  Select -ExpandProperty Lists | 
  Select -ExpandProperty Fields |
  Select TypeDisplayName -Unique|
  Sort TypeDisplayName

 

Here's the list I got from my test farm:

TypeDisplayName
---------------
All Day Event
Attachments
Audience Targeting
Calculated
Channel Alias
Check Double Booking
Choice
Computed
Content Type Id
Content Type ID
Counter
Cross Project Link
Date and Time
Event Type
File
Free/Busy
Guid
Hold Status
Hyperlink or Picture
Integer
Lookup
Managed Metadata
Moderation Status
Multiple lines of text
Number
Number of Likes
Number of Ratings
Out of Policy
Outcome choice
Page Separator
Permission Level
Person or Group
Publishing HTML
Publishing Image
Publishing Schedule End Date
Publishing Schedule Start Date
Rating (0-5)
Recurrence
Related Items
Resources
Single line of text
Summary Links
ThreadIndex
User Agent Substrings
Variations
Yes/No

 

.

Word Stemming in SharePoint 2013 Search

 

Word Stemming

Word stemming is when a search engine resolves a word to its root form and then returns variations of the word.

For example, search for one of these and return the rest:

• run, ran runner, running, runs
• fish, fishes, fisher, fishing

 

SharePoint 2013 Word Stemming

SharePoint 2013 only does "stemming" on singulars and plurals. So all we get for "run" is "runs" and for "fish" is "fishes". The exact stemming depends on the language.

SharePoint 2007 and 2010 did support an expanded form of word stemming, but you had to enable it in the search results web parts.

 

So what do we do?

Search using "OR".

   fish OR fishing OR fisher

   run OR ran OR running OR runs

Remember that the Boolean operators like "OR" must be typed in upper case and that mixtures of AND and OR usually will need parentheses.

   race AND (run OR ran OR running OR runs)

 

oh well…

 

.