SharePoint 2013 Search Weirdness – Part 1: What's a Duplicate? (or where's my file?)

 

This is part of the "Search Weirdness" series…

Can't find my purchase order!

I created several purchase orders and uploaded them to a library. I can see them in the library, and when I use the library's search box I still see all of them (seven).

 

Now I go to the search box at the top of the page and try to find the purchase orders:

 

But I can only find two of them! I find the same two if I go to the Enterprise Search Center site.

 

Although there were more than two files in the search results, only the above two of the seven purchase orders were found.

 

Duplicates!

Turns out Search thinks six of these files are duplicates. They are near duplicates… The first six Purchase Orders vary only by Purchase Order number and Order Date. Not an uncommon thing when I what to order the same thing several months in a row. What's special about the seventh? It has a note added that says "Please ship via truck".

 

How do you find duplicates?

There are two options to deal with duplicates that both require an edit to the Search Results web part.

Option 1: Give the user a link to request the duplicates.

Option 2: Just show all of the duplicates.

 

Option 1: Give the user a link to request the duplicates

To find duplicates you need to edit the Search Results web part. The only problem with that is that by default the search box at the top of site pages points to _layouts/15/osssearchresults.aspx, which is not customizable by site owners or at all in Office 365. But we can customize the search results pages in an Enterprise Search Center.

1. Go to your search results page (you will have at least five, but will at least want to edit the results.aspx page), edit the page, and edit the Search Results web part.
2. Check the Show View Duplicates Link box.
   
3. Click OK and save the page.
4. Add an update to your "Site customization documentation". (You do have this don't you? )

Now your users can click the "VIEW DUPLICATES" link, if they can find it!

 

View the Duplicates

1. Perform the search…
2. Ask yourself "where's my purchase order?" 
3. Mouse over each of the purchase orders in the search results.
4. In the hover panel (pop out) click VIEW DUPLICATES.
5. Still can't find your PO? Click the back button and mouse over the next file and repeat!

 

 

More Weirdness?

I had seven purchase orders. When I moused over PO 12351 and clicked VIEW DUPLICATES I found four purchase orders: 12346, 12347, 12348 and 12351. When I moused over 12350 and clicked VIEW DUPLICATES I found six purchase orders: 12345, 12346, 12345, 12348, 12349 and 12350.

 

Option 2: Just show all of the duplicates

1. Go to your search results page (you will have at least five, but will at least want to edit the results.aspx page), edit the page, and edit the Search Results web part.
2. Click the Change Query button.
3. Click the Settings tab.
4. Click "Don't remove duplicates".
    
5. Click OK, OK and then save the page.
6. Add an update to your "Site customization documentation". (You do have this don't you? )

Goodie! I can now find all of my purchase orders! All seven of them.

 

 

All Done?

You have updated your Enterprise Search Center. What about all of those site search boxes?

 

Option 1: Train all of your users to use the library's search box or the Enterprise Search Center whenever they can't find what they are looking for with the site search box.

Option 2: Visit every site and, manually or using PowerShell, update the Search Navigation to use the Enterprise Search Center for all searches. (another reason you need a Search Administrator!)

 

Bug or Feature?

You decide…

You may want to take a look at these:

http://blogs.technet.com/b/fesiro/archive/2013/11/11/sharepoint-2013-search-near-duplicates-and-documentsignature.aspx

http://blogs.perficient.com/microsoft/2013/04/sharepoint-2013-search-not-display-all-results/

 

 

Now to pick another weirdness for the next article…

.

SharePoint 2013 Search Weirdness

In my presentation at last week's SharePoint Cincy 2015 I had a slide titled "Search Weirdness" that raised a few eyebrows. It had a list of things that are, well, unexpected at the least. As I keep finding more of these "weird" things I thought I should put together a little series of blog articles on them.
Here they are:

1. Part 1: What's a Duplicate? (or where's my purchase order?)
2. Part 2: Can't Find My Videos!
3. Part 3: When do Daily and Weekly Search Alerts Get Sent?
4. Part 4: More Duplication Weirdness
5. Part 5: REST API Ignores Duplicates
6. Part 6:???

Also see:

• SharePoint Search Tip: Search for "title" where there is no "title"!
• SharePoint Search and Dates
• Word Stemming in SharePoint 2013 Search
• SharePoint 2013 Search Spelling Suggestions Confusion?
• SharePoint 2013 Search File Extensions
• Run SharePoint 2013 Search Reports from PowerShell

.

Run SharePoint 2013 Search Reports from PowerShell–Part 2!

 

First… see the original article here: http://techtrainingnotes.blogspot.com/2015/04/run-sharepoint-2013-search-reports-from.html

What if you want reports for each site collection? All we need is another function, a "Get-SPSIte –Limit ALL" and some code to generate unique file names and we can now create hundreds to thousands of Excel files!!! (Please make sure you never fill your server's drives with this!)

 

# This is the path to write the reports to:
$path = "c:\SearchReports\"


function Get-SPSearchReports ($farmurl, $searchreport, $path)
{
  # Report names and IDs
  $Number_of_Queries          = "21be5dff-c853-4259-ab01-ee8b2f6590c7"
  $Top_Queries_by_Day         = "56928342-6e3b-4382-a14d-3f5f4f8b6979"
  $Top_Queries_by_Month       = "a0a26a8c-bf99-48f4-a679-c283de58a0c4"
  $Abandoned_Queries_by_Day   = "e628cb24-27f3-4331-a683-669b5d9b37f0"
  $Abandoned_Queries_by_Month = "fbc9e2c1-49c9-44e7-8b6d-80d21c23f612"
  $No_Result_Queries_by_Day   = "5e97860f-0595-4a07-b6c2-222e784dc3a8"
  $No_Result_Queries_by_Month = "318556b1-cabc-4fad-bbd5-c1bf8ed97ab1"
  $Query_Rule_Usage_by_Day    = "22a16ae2-ded9-499d-934a-d2ddc00d406a"
  $Query_Rule_Usage_by_Month  = "f1d70093-6fa0-4701-909d-c0ed502e3df8"

  # create a unique filename for each site and report
  $sitename = $farmurl.Replace("http://","").Replace("https://","")
  $sitename = $sitename.substring(0,$sitename.IndexOf("/_layouts"))
  $sitename = $sitename.Replace("/","_").Replace(":","_").Replace(".","_")

  $filename = $path + $sitename + " " + 
              (Get-Variable $searchreport).Name + " " + 
              (Get-Date -Format "yyyy-mm-dd") + ".xlsx"
  #Write-Host "Creating $filename"

  $reportid = (Get-Variable $searchreport).Value

  $TTNcontent = "&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D" + $reportid

  # setup the WebRequest
  $webRequest = [System.Net.WebRequest]::Create($farmurl)
  $webRequest.UseDefaultCredentials = $true
  $webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*"
  $webRequest.ContentType = "application/x-www-form-urlencoded"
  $webRequest.Method = "POST"

  $encodedContent = [System.Text.Encoding]::UTF8.GetBytes($TTNcontent)
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()

  # get the data
  [System.Net.WebResponse] $resp = $webRequest.GetResponse();
    $rs = $resp.GetResponseStream();
    #[System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    #[byte[]]$results = $sr.ReadToEnd();
    [System.IO.BinaryReader] $sr = New-Object System.IO.BinaryReader -argumentList $rs;
    [byte[]]$results = $sr.ReadBytes(10000000);

  # write the file
  Set-Content $filename $results -enc byte
}


function Get-SPSearchReportsForSite ($url, $path)
{

  Get-SPSearchReports $url "Number_of_Queries" $path
  Get-SPSearchReports $url "Top_Queries_by_Day" $path
  Get-SPSearchReports $url "Top_Queries_by_Month" $path
  Get-SPSearchReports $url "Abandoned_Queries_by_Day" $path
  Get-SPSearchReports $url "Abandoned_Queries_by_Month" $path
  Get-SPSearchReports $url "No_Result_Queries_by_Day" $path
  Get-SPSearchReports $url "No_Result_Queries_by_Month" $path
  Get-SPSearchReports $url "Query_Rule_Usage_by_Day" $path
  Get-SPSearchReports $url "Query_Rule_Usage_by_Month" $path

}


Get-SPSite -Limit All | 
  foreach   {
    $url = $_.Url + "/_layouts/15/Reporting.aspx?Category=AnalyticsSiteCollection";
    #$path = (you might do a custom path for each set of reports)

    Write-Host "Getting files for $url"
    Get-SPSearchReportsForSite $url $path
  }

Run SharePoint 2013 Search Reports from PowerShell

Update: SharePoint 2016 version here: http://techtrainingnotes.blogspot.com/2018/02/run-sharepoint-2013-and-2016-search.html



Update! Need these reports for every site collection in the farm? See Part 2: http://techtrainingnotes.blogspot.com/2015/04/run-sharepoint-2013-search-reports-from_21.html

In my Search Administration class I stress that admins should dump the search reports on a regular basis as the data is only kept in detail for 14 days and in summary form for 35 months. But who wants to both run these reports at least once every 14 days, even they can remember to do so. So, PowerShell to the rescue… Schedule this script to run each weekend and your work is done.
The following script works for on premise SharePoint 2013. To work with Office 365 you will have to figure out how to include your credentials. The example included here works on premises by using "UseDefaultCredentials = $true".
After lots of hacking, detective work (see below) and just plain trial and error, here's the script:

# This is the URL from the Central Admin Search Service Usage Reports page:
$url = "http://yourCentralAdminURL/_layouts/15/reporting.aspx?Category=AnalyticsSearch&appid=ed39c68b%2D7276%2D46f7%2Db94a%2D4ae7125cf567"  

# This is the path to write the reports to (must exist):
$path = "c:\SearchReports\"




function Get-SPSearchReports ($farmurl, $searchreport, $path)
{
  # TechTrainingNotes.blogspot.com

  # Report names and IDs
  $Number_of_Queries          = "21be5dff-c853-4259-ab01-ee8b2f6590c7"
  $Top_Queries_by_Day         = "56928342-6e3b-4382-a14d-3f5f4f8b6979"
  $Top_Queries_by_Month       = "a0a26a8c-bf99-48f4-a679-c283de58a0c4"
  $Abandoned_Queries_by_Day   = "e628cb24-27f3-4331-a683-669b5d9b37f0"
  $Abandoned_Queries_by_Month = "fbc9e2c1-49c9-44e7-8b6d-80d21c23f612"
  $No_Result_Queries_by_Day   = "5e97860f-0595-4a07-b6c2-222e784dc3a8"
  $No_Result_Queries_by_Month = "318556b1-cabc-4fad-bbd5-c1bf8ed97ab1"
  $Query_Rule_Usage_by_Day    = "22a16ae2-ded9-499d-934a-d2ddc00d406a"
  $Query_Rule_Usage_by_Month  = "f1d70093-6fa0-4701-909d-c0ed502e3df8"


  $filename = $path + (Get-Variable $searchreport).Name + " " + (Get-Date -Format "yyyy-mm-dd")  + ".xlsx"
  $reportid = (Get-Variable $searchreport).Value

  $TTNcontent = "&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId%3D" + $reportid

  # setup the WebRequest
  $webRequest = [System.Net.WebRequest]::Create($farmurl)
  $webRequest.UseDefaultCredentials = $true
  $webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, */*"
  $webRequest.ContentType = "application/x-www-form-urlencoded"
  $webRequest.Method = "POST"

  $encodedContent = [System.Text.Encoding]::UTF8.GetBytes($TTNcontent)
    $webRequest.ContentLength = $encodedContent.length
    $requestStream = $webRequest.GetRequestStream()
    $requestStream.Write($encodedContent, 0, $encodedContent.length)
    $requestStream.Close()

  # get the data
  [System.Net.WebResponse] $resp = $webRequest.GetResponse();
    $rs = $resp.GetResponseStream();
    #[System.IO.StreamReader] $sr = New-Object System.IO.StreamReader -argumentList $rs;
    #[byte[]]$results = $sr.ReadToEnd();
    [System.IO.BinaryReader] $sr = New-Object System.IO.BinaryReader -argumentList $rs;
    [byte[]]$results = $sr.ReadBytes(10000000);

  # write the file
  Set-Content $filename $results -enc byte
}



Get-SPSearchReports $url "Number_of_Queries" $path
Get-SPSearchReports $url "Top_Queries_by_Day" $path
Get-SPSearchReports $url "Top_Queries_by_Month" $path
Get-SPSearchReports $url "Abandoned_Queries_by_Day" $path
Get-SPSearchReports $url "Abandoned_Queries_by_Month" $path
Get-SPSearchReports $url "No_Result_Queries_by_Day" $path
Get-SPSearchReports $url "No_Result_Queries_by_Month" $path
Get-SPSearchReports $url "Query_Rule_Usage_by_Day" $path
Get-SPSearchReports $url "Query_Rule_Usage_by_Month" $path
 

The Detective Work…

I could not find anything documented on how the reports are called or details on things like the report GUIDs. So here's how I got there:

• Go the search reports page in Central Admin and press F12 to open the Internet Explorer F12 Developer Tools then:
  • Click the Network tab and click the play button to start recording.
  • Click one of the report links.
  • Double-click the link generated for the report in the F12 pane to open up the details.
  • Make note of the URL (It's the same as the report page!)
  • Note the Accept, and Content-Type Request Headers.
  • Click the Request Body tab.
  • Stare at 3000 characters in that string until your head really hurts, or until you recognize most of what is there is the normal page postback stuff like VIEWSTATE. So we need to find what's unique in the string. (It's the Report IDs.)
  • Click on each of the nine reports and copy out the report IDs.
  • With a lot of trial and error figure out what the minimum string needed is to generate the reports. (It's ""&__EVENTTARGET=__Page&__EVENTARGUMENT=ReportId" plus the report id.)
  • Find out how to do an HTTP POST using PowerShell. (Steal most of it from here: http://www.codeproject.com/Articles/846061/PowerShell-Http-Get-Post.)
  • Find some other needed .Net code and convert the C# to PowerShell.
  • Fill in some gaps with PowerShell putty …….

.

SharePoint 2013 Items Removed with Search Result Removal Return from the Dead!

 

In SharePoint 2010, Removed was Removed!

In SharePoint 2010 there's a search feature called "Search Result Removal" that made an item immediately disappear from user's search results. All you had to do was enter the URL to the problem item and click Remove Now. SharePoint then remove the item from the index and wrote a Crawl Rule to make sure the content was ignored by the crawler in all future searches.

The 2010 request to remove:

 

The auto-created Crawl Rule:

SharePoint 2010 had a interesting bug here. Uppercase letters in the URL would cause the the removal request to be ignored!

 

In SharePoint 2013 and Office 365, Removed is Just a Temporary Thing! Maybe only a few seconds!

We don't have Crawl Rules in SharePoint 2013 and Office 365. When you request the removal of an item or a URL from the search index, it just deletes it from the index. But… on the next crawl it adds it back! That next crawl could be an hour away, or only seconds, depending on your crawl schedules and the luck of your timing.

Nowhere can I find any documentation that says this!

The screen for removal requests is similar to what we had with 2010, but with no mention about Crawl Rules.

 

Removal Straight from the Crawl Logs

Both versions let you browse the crawl logs, find a problem item, click the dropdown on the item and click Remove the Item From The Index. In the Crawl Logs you can remove one item at a time while the Removal pages will let you exclude and entire path.

In 2013 we are just told that the item will be removed. In 2010 we are also told that a crawl rule will be created.

 

References:

Remove URLs from search results (SharePoint Server 2010
https://technet.microsoft.com/en-us/library/ff621095(v=office.14).aspx

Delete items from the search index or from search results in SharePoint Server 2013
https://technet.microsoft.com/en-us/library/jj219587.aspx

SharePoint Search Tip: Search for "title" where there is no "title"!

 

Title:somekeywords

You find the strangest things in SharePoint when you are looking for something else! I'm writing the labs for a new SharePoint 2013 Search Administration class and was looking for a list that did not support a search on "title:sometext". The list I first tested that did not have a "Title" column actually did work with a search for "title:". Huh? So I next tried the Links List as this one has always been a problem with the Content Query Web Part due to not having a "Title" column. I found that I could search Links with "title:sometext"! How?

 

Everything has a Title?

Almost…

Most SharePoint lists and libraries have a "Title" column. As "Title" is also an out of the box search Managed Property you can search these lists and libraries by using "title:airplane". If you browse around your sites though you will only find a few lists and libraries that have a "Title" column. For example: document libraries and announcements lists.

Out of the box lists and libraries that support a search on Title:

• Some libraries and lists have an obvious Title column, so they are searchable on "Title". While libraries have a Title column, it is often not populated by users. Search will index the "Title" as the title field, when not blank, and otherwise index the "Name" column as "Title", so we can still search using "title:keyword".
• Asset libraries have a "Title" column listed in the Library Properties page, but it's not listed on the Edit Properties page of videos, but we can still search using "title:keyword".
• Custom lists where you have renamed the default title. Even after being renamed, it's still "Title" internally.
• Lists with no displayed title column:
  • A Contacts list's "Last Name" column is internally named "Title". I.e. a search for title:smith works but a search for title:"mike smith" does not as "Mike" is not in the "Title/Last Name" column.
  • A Task list's "Task Name" column is internally named "Title".
  • A Discussion list's "Subject" column is internally named "Title".
  • A Categories list's "Category" column is internally named "Title".
  • A Content and Structure Reports list's "Report Title" column is internally named "Title".
  • A Links list has a hidden Title column that is auto-populated with the text from the "description" of the link.
  • A list created by importing an Excel file makes the first text column the clickable link column for the list. While the displayed name is whatever was imported, the internal name is "Title".

What does not have a Title?

• Site Pages libraries.
• Wiki libraries.
• Visio Repository Site Process Diagrams libraries.
• An external list (External Content Type) may not have a "Title" column.

 

Bottom line?

Almost every list is searchable using "title:somekeywords".

 

 

About the Links List

A Links list is special… or a least different. While many lists have a visible column with a name other than "Title" and an internal name of "Title", the List list is a bit different. A links list does not seem to have a end user accessible Title column, which is why its always been a pain with the Content Query Web Part. But… it does have a "Title" column, it's just hidden and Read Only. If you do a search using "title:", you will find links!  I guess its auto-populated and included just for searching. Here's the XML from the feature for the Links list:

 

.

A note to self-published book authors…

 

Actually a few notes…

 

I found two interesting SharePoint 2013 search books at Amazon and ordered them. I just received them… and I'm going to return both of them. This will be the first time I have ever returned a printed book!

 

Self-publishing is easy, fun and very rewarding. But…

Note 1: All printed technology books must have an index! Indexes are easy to create in Microsoft Word. You can either create a busy looking, but not too useful, index using the automatic features, or invest a couple hours and manually create a very useful one. In any case be forewarned, I do not buy technical books that do not have an index!

Note 2: Do not tell me in the book's intro that all of the content in the book is straight from your blog articles. Especially don't tell me that I could have gotten it all for free from the blog. (Boy, do I feel dumb paying for that book.) Don't just copy your blog articles and call it a book. Books that build on your blog articles can really be useful. Creating the book is your opportunity to expand on the topic and rewrite the quickly written text into nice complete articles. Fix up the screen captures. Fill in the gaps. Rewrite the hastefully written text. You know… create what you would have done in the blog if you had time to do it right the first time.

Note 3: Do not fill the pages with giant screen captures so there's an average of one screen capture and only one or two sentences per page. That works for a kid's picture book, but does not deliver much value as a technical book. In any case, please, please size and crop the screen captures so they are always readable and to the point.

Note 4: Pay somebody, anybody, to proofread the book. They don't have to know the technology. They just have to know some basic English and know enough to ask if all the code really should be underlined! If you want to really do it right, find someone who has heard of the Chicago Manual of Style, At a minimum buy yourself a copy of the Microsoft Manual of Style, and give a copy to your proofreader.

Note 5: Look at some of the tech books from the big publishers. I mean, open them, look at the details, note what is there and what is not. Note the use of "front matter". Note that page numbers go on the outside edges of pages, not in the gutter. Note the use of font size and the limited use of bold and italics.

 

</rant>

 

.

How do you spell SharePoint?

 

Ok, this is useless trivia… In the list of SharePoint 2013 Search Result Types (Settings, Site Settings, Search section, Result Types) it looks like someone at Microsoft is not sure how to spell SharePoint! At least they did not spell it SharePint!

Oh wait… they made up for it in the next word. Everywhere in TechNet microblog has a lowercase "b"!

So, for accuracy, in my courseware should I use "Sharepoint MicroBlog", "SharePoint MicroBlog" or "SharePoint Microblog"?   Decisions, decisions, decisions… Oh the life of a tech writer…

 

.