8/04/2020

Merging JSON files using PowerShell


Let's say you have a directory full of JSON files and you need to merge them into a single file.


Emp1.json

    {
        "Name":  "Mike",
        "ID":  123,
        "Status":  "Active"
    }

Emp2.json

    {
        "Name":  "Susan",
        "ID":  124,
        "Status":  "Active"
    }

Emp3.json

    {
        "Name":  "Sam",
        "ID":  125,
        "Status":  "Inactive"
    }

You need a result that includes commas and square brackets that looks like this:

[
    {
        "Name":  "Mike",
        "ID":  123,
        "Status":  "Active"
    },
    {
        "Name":  "Susan",
        "ID":  124,
        "Status":  "Active"
    },
    {
        "Name":  "Sam",
        "ID":  125,
        "Status":  "Inactive"
    }
]



Solution 1:

Combing the files using Get-Content and append the start and end brackets.

"[" + ((Get-Content Emp*.json -raw) -join ","  ) + "]" | Out-File AllEmp.json

Note that the "-raw" reads the entire file as a single string, not an array of strings. That's how we can add a comma between each block of JSON. So, each file is read as a single string, each string is added by Get-Content to an array. The array of JSON strings is then joined into a single long string with a comma between each JSON string.

You might want the brackets on their own lines, so add some line breaks.

"[`n" + ((Get-Content Emp*.json -raw) -join ","  ) + "`n]" | Out-File AllEmp.json

What if you wanted to sort them or manipulate the objects' properties as they are merged? You will need to convert them to objects first, then sort or manipulate the objects, and then convert them back to JSON.

( "[" + ((Get-Content Emp*.json -raw) -join ","  ) + "]"  | ConvertFrom-Json ) | sort name | ConvertTo-Json | Out-File AllEmp.json


What if you have a lot of files? The examples above bring all of the files into memory and create one large string. Here's a solution that reads each file and appends to the merged file.
  • The first line creates the new file and writes out the starting bracket.
  • The second line creates our delimiter and sets it "" for the first item.
  • The third line gets each JSON file and writes it to the new file with a leading delimiter. After the first file it sets the delimiter to ",".
  • The last line adds the closing square bracket.

"[" | Out-File AllEmps.json -Force
$delim = "";
Get-ChildItem Emp*.json | foreach { $delim + (Get-Content $_ -raw); $delim="," } | Out-File AllEmps.json -Append }
"]" | Out-File AllEmps.json -Append


8/03/2020

How to count lines of code using PowerShell

I've got a project... scattered through a number of folders are a bunch of C# files. How many lines of code do I have? (Bragging rights you know...)

You could open the project in Visual Studio and use the Analyze, Calculate Code Metrics. But that requires Visual Studio, and does not give me all the options I want. (But in many ways is better! Lines by file and by method are available there.)

I wanted to do this on raw files using PowerShell, so assuming your project is in C:\myProject and you only wanted to count ".cs" files...

  dir C:\myProject -Recurse *.cs | Get-Content | measure

Get-Content returns the lines from the file as an array, and Measure-Object counts things.

Want to count characters too?

  dir C:\myProject -Recurse *.cs |
    Get-Content |
      measure
-Line -Character

Your boss wont let you take credit for blank lines?

  dir C:\myProject -Recurse *.cs |
    Get-Content |
      where { $_.trim() -ne "" } |
        measure
-Line -Character

Your boss wont let you take credit for blank lines or comments? (I would pay extra for good comments!)

  dir C:\myProject -Recurse *.cs |
    Get-Content |
      where { $_.trim() -ne "" -and $_.trim() -notlike "//*" } |
        measure
-Line -Character

    (Sorry, but in this quick little exercise I did not deal with /* and */ block comments, only // comments.)

Want to include multiple file types?

  dir C:\myProject -Recurse -Include *.cs, *.cshtml |
    Get-Content |
      where { $_.trim() -ne "" -and $_.trim() -notlike "//*" } |
        measure
-Line -Character


Boy I've been working hard... and using wizards to write code... 😉

   Lines Words Characters Property
   ----- ----- ---------- --------
    2563           117084

4/25/2020

PowerShell: When does zero equal one? (Length property of a Directory)


You learn the most interesting things when teaching a class, especially when an off the cuff demo goes wrong.

During a simple demo of a calculated or derived column I did this:

For files, the output was as expected:


But for directories it returned this:


While the FileInfo object does have a Length property, the DirectoryInfo object does not. So what should we expect when both files and directories are in the same pipeline? Directories usually display a blank column for Length. Is this a Null? Is this an empty or one space string? Or is it just skipped because DirectoryInfo objects don't have a Length property?

Noting that 9.53674E-07 is 1 / 1MB, PowerShell is returning a 1 for the missing Length property, and not a zero or null as I was expecting.

Turns out that Length is both a property of FileInfo objects, and also a property of all PowerShell objects. In my example, DirectoryInfo objects did not have a Length property so $_.Length returns the underlying object's Length property.

Here's an example of a new "Object". Note that PowerShell thinks it has a length of 1 even though Length is not one of its properties. If it's not a defined property, where does it come from? PowerShell seems to treat all objects as collections, even if it is a single item. Collections have both a Count and a Length property. So, $x has a Length (and a Count) of 1.


There's a hint in the PowerShell help files about this. https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_properties?view=powershell-7  (or in PowerShell help about_Properties)



And, as the Help file says, it was different in PowerShell 2.0. If I launch PowerShell with the -Version 2.0 option, I actually get the result I expected in the original demo!





.

4/01/2020

SharePoint vs. Teams vs. OneDrive



A few notes from a recent discussion about where to store files:

 
  • SharePoint Everywhere:
    • SharePoint is behind OneDrive, SharePoint and Teams. Each exposes a different level of functionality.
    • In Teams you can display as “Teams Files”, or click a link and open the backing SharePoint site. In OneDrive, most SharePoint functionality is hidden.
    • SharePoint sites can have multiple libraries (similar to multiple drive letters/network shares) while OneDrive and Teams only expose a single library.
  • Content Location and Ownership:
    • OneDrive is associated with individual users, and the content is “owned” by individuals. (OneDrive is actually SharePoint, but most of the SharePoint functionally is hidden.)
    • SharePoint libraries are associated with a SharePoint Site and the content is “owned” by the Site Owners, who are delegated to that role and can be quickly replaced in the future.
    • Teams file storage is in a disguised SharePoint library. It is “owned” by the Team owners.
  • Content Sharing:
    • OneDrive “sharing” is somewhat “ad hoc” and not driven by corporate policy or best practices. OneDrive users can share with anyone.
    • SharePoint “sharing” can be “ad hoc”, but is usually managed by assigning permissions to users, much like network shares. If done right, SharePoint allows quick and easy auditing of “who has what access to this file”.
    • Teams content by default is shared with the team.
  • Content Sync
    • The default use of OneDrive is via the “Sync” feature. Multiple users editing/deleting sync’d content impacts all users syncing that content.
    • The default use of SharePoint and Teams is closer to network shares. Files are stored there, and downloaded, or viewed/edited online, as needed. Libraries can be mapped to local drive letters. We typically discourage any SharePoint library syncing.


Governance is very important, especially with OneDrive. Some things to consider:
  • Who “owns” a file? Storing the file in OneDrive implies that the user owns it. Storing it in Teams or SharePoint implies the organization owns it. If in OneDrive, what’s the impact if a user leaves the company or the department? If in Teams, what’s the impact if a Team is deleted?
  • Who should grant access to corporate content? A OneDrive user can share with anyone. It’s so easy, the user rarely is thinking about security. In SharePoint or Teams, an “owner” grants and removes permissions for users and can easily update and audit user access.
  • Which copy of a document is the “single source of truth”? If we each have a copy in our OneDrive or local drive, which one is the official version?
  • If data with a legal impact is stored, who should manage it? Each user with their OneDrive, a “content steward” who manages a SharePoint site? A Teams owner (who has been property trained on your governance)?



  •  

Note to spammers!

Spammers, don't waste your time... all posts are moderated. If your comment includes unrelated links, is advertising, or just pure spam, it will never be seen.