The Classy PlatyPS: Automated Class, Enum, and Private Function Documentation for PowerShell Module CI/CD Pipelines

 Picture source: http://newproductions.deviantart.com/art/platypus-even-more-gentlemanly-261415442

Introduction

I have spent the past month working feverishly to release the core functionality of my PSRAW module. I started several blog posts to try and cover some of the things I was doing as I was doing them, but I ended up changing directions so often and quickly that trying to "blog as I go" became a wasted effort. But, I’m really excited about much of what the project has shaped up to be.

One feature of the project I am most proud of is what I’m calling "The Classy PlatyPS" auto-documentation feature. This is basically a total revamp of the auto-documentation system in my 4 Part Blog series. I planned to do another series on the entire new pipeline, and I likely will, but I’m just too excited about this feature to wait!

What is The Classy PlatyPS?

As a quick review, PlatyPS is a PowerShell module that makes it easy to maintain module help as markdown. This works great for your module's public functions but offers nothing in the way of generating and maintaining documentation for your module’s Classes, Enums, or Private Functions. It does, however, offer the ability to create and integrate about_ topics into your help system. But this would be a manual step to generate an about_ topic for each Class and Enum and then another manual task to update that topic when you add or remove properties, methods, constructors and fields.

The Classy PlatyPS adds automatic about_ topics for the Classes and Enums in your module. It also will add and remove sections for properties, methods, constructors, and fields as you add and remove the same from your Classes and Enums. It works in concert with PlatyPS so that these markdown files are added to the modules and thus available via Get-Help.  It also adds and maintains online documentation for Module Private Functions via PlatyPS.

On Module Classes

This really deserves a blog post of its own. I have started such a blog post, but it is more of a series because there is just so many possibilities and issues surrounding Classes in modules. So, rather than go into all the intricacies here, I will just promise that at some point I will deliver a post/series on this topic at length and that you will just have to take my word here for now.

For now, what you need to know is that The Classy PlatyPS requires a “new” way of using Classes and Enums in your module. First, the Class and Enum definitions must be in separate .ps1 files. The Class definition files will also need to begin with a 3-digit number to indicate its load order. Lower numbers will be loaded first so if on class depends on another, the dependency needs to be loaded first. Second, the Class and Enum files need to be “exported” by the module by adding them to the ScriptsToProcess setting in the module manifest. Finally, they also need to dot sourced in the .psm1.

Folder Layout:

.\PSRAW │ PSRAW.psd1 │ PSRAW.psm1 │ ├───Classes │ 001-RedditOAuthScope.ps1 │ 002-RedditApplication.ps1 │ 003-RedditOAuthCode.ps1 │ 004-RedditOAuthToken.ps1 │ 005-RedditApiResponse.ps1 │ └───Enums RedditApplicationType.ps1 RedditOAuthDuration.ps1 RedditOAuthGrantType.ps1 RedditOAuthResponseType.ps1

PSRAW.psd1:

# ...Snip ScriptsToProcess = @( 'Enums\RedditApplicationType.ps1', 'Enums\RedditOAuthDuration.ps1', 'Enums\RedditOAuthGrantType.ps1', 'Enums\RedditOAuthResponseType.ps1', 'Classes\001-RedditOAuthScope.ps1', 'Classes\002-RedditApplication.ps1', 'Classes\003-RedditOAuthCode.ps1', 'Classes\004-RedditOAuthToken.ps1', 'Classes\005-RedditApiResponse.ps1' ) #Snip...

PSRAW.psm1:

# ...Snip $functionFolders = @('Enums', 'Classes', 'Public', 'Private' ) ForEach ($folder in $functionFolders) { $folderPath = Join-Path -Path $PSScriptRoot -ChildPath $folder If (Test-Path -Path $folderPath) { Write-Verbose -Message "Importing from $folder" $FunctionFiles = Get-ChildItem -Path $folderPath -Filter '*.ps1' -Recurse | Where-Object { $_.Name -notmatch '\.tests{0,1}\.ps1' } ForEach ($FunctionFile in $FunctionFiles) { Write-Verbose -Message " Importing $($FunctionFile.BaseName)" . $($FunctionFile.FullName) } } } # Snip..

One cool thing about this method is that the Classes and Enums will be automatically imported into the calling scope when Import-Module is called. Again, I plan to cover this more in depth in another blog post/series as this too is another feature I'm pretty excited about. but it's necessary to understand that for The Classy PlatyPS to do its magic, you need to use this method.

The Classy PlatyPS BuildDocs

The Classy PlatyPS Believes Markdown is the One True God

If you recall from my 4-part series on auto-documentation, I was using a psake task named BuildDocs. I discovered some really interesting bugs while trying to use the same process from PSMSGraph (which used the dynamic type system instead of classes). First, I discovered that PlatyPS was not meant to use the Comment Based Help as the source of truth. With PlatyPS you are supposed to use the markdown as the source of truth. My first order of business was to adjust my process around that. You will notice a complete absence of Comment Based Help from function definitions. This is necessary if you want the external help generated by PlatyPS to work. Comment Based Help trumps External Help so it's presence causes some maintenance issues.

The Classy PlatyPS Gets a Job

The next bug I ran into is that PlatyPS as some weird scope issues where classes defined in the module aren't visible when it is parsing the module functions, but only when it is called from a psake task. I don't even know where to begin with this issue, but the gist of it is that since PlatyPS is ignorant of the classes it starts to throw errors trying to process arguments for the public functions that take the module classes as parameters. Again, this only happens within the psake context. if you run the same exact code as a standalone script it works fine.

The solution, then, was to create a separate script and run int as a job from psake. Dot sourcing, & invoking and Invoke-Command all have the same weird scope issue for which I have no explanation. But, Jobs run in a completely separate PowerShell context so we now call the separate BuildDocs.ps1 script as a job.

psake.ps1:

# ...Snip Task BuildDocs -depends CodeCoverage { $lines Start-Job -FilePath "$ProjectRoot\BuildTools\BuildDocs.ps1" -ArgumentList @( $env:BHPSModuleManifest $ModuleName $MkdcosYmlHeader $ChangeLog $ProjectRoot $ModuleFolder $ReleaseNotes $true $true $true ) | Wait-Job | Receive-Job "`n" } # Snip...

Putting the "Class" in The Classy PlatyPS

After I had the BuildDocs process separated out I wanted to get the auto-documentation of Classes and Enums added to the process. The first problem I needed to solved was just how you figure out what Classes and Enums are being exported by the module. Like most things related to PowerShell v5 classes, there is no documentation on this and I was pioneering new territory.

After some digging around on the types exported from my module I discovered that there is something a bit unique to PowerShell v5 classes from regular .NET classes. It's hard to explain, so here is the RedditApplicationType Enum from my module as an example:

PS C:\> [redditApplicationType].Assembly.Modules[0]
MDStreamVersion : 131072 FullyQualifiedName : ModuleVersionId : 7b468bd7-15ff-4669-bbac-1213a8e12afb MetadataToken : 1 ScopeName : ⧹C։⧹PSRAW⧹PSRAW⧹Enums⧹RedditApplicationType.ps1 Name : Assembly : ⧹C։⧹PSRAW⧹PSRAW⧹Enums⧹RedditApplicationType.ps1, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null CustomAttributes : {} ModuleHandle : System.ModuleHandle

You can see that the ScopeName has a funky version of the script path where the Enum is defined. if you look at the scope name of other .NET classes, you get something closer to the dll they are in:

PS C:\> [datetime].Assembly.Modules[0]
MDStreamVersion : 131072 FullyQualifiedName : C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscorlib.dll ModuleVersionId : d401a7e1-a31e-47e3-87e3-cb92c12b437f MetadataToken : 1 ScopeName : CommonLanguageRuntimeLibrary Name : mscorlib.dll Assembly : mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 CustomAttributes : {[System.Security.UnverifiableCodeAttribute()]} ModuleHandle : System.ModuleHandle

PS C:\> [System.Web.HttpUtility].Assembly.Modules[0]
MDStreamVersion : 131072 FullyQualifiedName : C:\WINDOWS\Microsoft.Net\assembly\GAC_64\System.Web\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Web.dll ModuleVersionId : f914f5be-9ef1-4be0-afc3-3c88d39da087 MetadataToken : 1 ScopeName : System.Web.dll Name : System.Web.dll Assembly : System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a CustomAttributes : {[System.Security.UnverifiableCodeAttribute()]} ModuleHandle : System.ModuleHandle

This means that it is possible to find the module Classes and Enums using reflection. This is great! This means that not only can we detect the module classes for the purposes of documenting them, but also for testing. I created a Get-ModuleClass function to help with this:

function Get-ModuleClass { [CmdletBinding()] param ( [Parameter( Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true )] [string[]] $ModuleName ) process { foreach ($Name in $ModuleName) { $Module = $null Write-Verbose "Processing Module '$Name'" $Module = Get-Module -Name $Name -ErrorAction SilentlyContinue if (-not $Module) { Write-Error "Module '$Name' not found" continue } [System.AppDomain]::CurrentDomain. GetAssemblies(). gettypes(). where( {$_.Assembly.Modules[0].ScopeName -match "$Name" -and $_.IsPublic }) } } }

You supply the module name and it returns all the Classes and Enums associated with the module. This does require that export method I detailed before is used and that the module is already imported in the calling scope.

The Classy PlatyPS Embraces Change

Need to add a new Class or Enum? Need to add a new Property, Method, Constructor, or Field? Sick of manually documenting all those changes? Never fear, The Classy PlatyPS will cover all those bases.

The Classy PlatyPS Creates Templates for New Classes

PlatyPS includes a template for about_ topics. This became the template for my templates. I had originally planned to use Plaster for this, but, there are too many moving parts for Plaster to handle. Using reflection, The Classy PlatyPS looks at Classes and Enums and and generates a PlatyPS compatible Markdown template containing entries for all Properties, Methods, and Constructors. It provides information for them such as what their data types are, whether they are hidden or static, definitions, etc. It looks very similar to MSDN documentation for .NET classes. The Best Example of this is the RedditApplication class.

The code for the Reflection is a bit too lengthy to post here. For the full code see BuildDocs-Helper.ps1. As an example here is the MethodText function:

Function MethodText { [cmdletbinding()] param( [Parameter( Mandatory = $true, Position = 0, ValueFromPipeline = $true )] [Object[]] $Method ) process { foreach ($MyMethod in ($Method | Sort-Object Name ) ) { $Params = ($MyMethod.GetParameters() | ForEach-Object { $Type = $_.ParameterType -replace '^System\.([^.]*)$', '$1' "{0} {1}" -f $type, $_.name }) -Join ", " $Access = '' $IsHidden = $False if ($MyMethod.CustomAttributes.AttributeType.Name -contains 'HiddenAttribute') { $Access = 'hidden ' $IsHidden = $true } $IsStatic = $MyMethod.IsStatic $Scope = '' if ($MyMethod.IsStatic) { $Scope = 'static ' } $Name = $MyMethod.Name $ReturnType = $MyMethod.ReturnType.FullName -replace '^System\.([^.]*)$', '$1' $Definition = "{0}{1}{2} {3}({4})" -f $scope, $Access, $ReturnType, $Name, $Params $Heading = MethodHeading $MyMethod $ExecutionContext.InvokeCommand.ExpandString($MethodTemplate) } } }

It takes a System.Reflection.RuntimeMethodInfo object. You can get the methods from a class with something like this:

$Class = [RedditApplication] $Methods = $Class.GetMethods() | Where-Object {$_.IsSpecialName -eq $false}

There is some logic in place to sort methods, properties, fields, constructors alphabetically and by number of arguments.

The Classy PlatyPS Adds and Removes Documentation as Code Changes

This piece is new. I added it after the PSRAW release, but I began working on it a few weeks ago. I wanted to make it easy to keep the Class and Enum documentation up to date as I make changes. Some of these classes are still not in their final form and will have more properties and methods added as the project grows. The problem, however, is parsing markdown

The Classy PlatyPS Puts the Smack-down on Markdown

I first thought of approaching this problem with re-engineering PlatyPS as it contains code to convert Markdown to something code-readable. But after about 3 hours in the PlatyPS code I realized I was in over my head and it would be quicker to use either a ready-made PowerShell Markdown parsing module or a C# dll. My searches revealed that while there are tools out there, they all lack documentation. So, it was time to make my own purpose-built markdown parsing engine.

I created a ConvertFrom-Markdown function which returns a PowerShell object representation of the Markdown file. It's not an overly complex one like HTML DOM or XML. It just nests headings and sub headings and the inner text.

function ConvertFrom-MarkDown { [CmdletBinding()] param ( [Parameter(Mandatory = $true, Position = 0, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true, HelpMessage = "Path to one or more locations.")] [Alias("PSPath")] [ValidateNotNullOrEmpty()] [string[]] $Path ) process { foreach ($MyPath in $Path) { $Object = [System.Collections.Generic.List[System.Collections.Hashtable]]::new() $Headingindex = -1 Get-Content -Path $MyPath | ForEach-Object { $Line = $_ if ($Line -match '^\s*#[^#]') { $Object.add(@{ Heading = $Line -replace '^\s*#\s*' Text = [System.Collections.Generic.List[System.String]]::new() Subheadings = [System.Collections.Generic.List[System.Collections.Hashtable]]::new() }) $SubheadingIndex = -1 $Headingindex++ return } if ($Line -match '^\s*##[^#]') { $Object[$Headingindex].Subheadings.add(@{ Heading = $Line -replace '^\s*##\s*' Text = [System.Collections.Generic.List[System.String]]::new() }) $SubheadingIndex++ return } if ($SubheadingIndex -ge 0) { $Object[$Headingindex].Subheadings[$SubheadingIndex].Text += $Line return } $Object[$Headingindex].Text += $Line } $Object } } }

If you had the following Markdown

# Heading1 Heading1 text
## Subheading1 subheading1 text
## Subheading2 subheading2 text
# Heading2 Heading2 text
## Subheading3 subheading3 text
## Subheading4 subheading4 text

It would create the following PowerShell object

@( @{ Heading = 'Heading1' Text = 'Heading1 text' Subheadings = @( @{ Heading = 'Subheading1' Text = 'subheading1 text' Subheadings = @() } @{ Heading = 'Subheading2' Text = 'subheading2 text' Subheadings = @() } ) } @{ Heading = 'Heading2' Text = 'Heading2 text' Subheadings = @( @{ Heading = 'Subheading3' Text = 'subheading3 text' Subheadings = @() } @{ Heading = 'Subheading4' Text = 'subheading4 text' Subheadings = @() } ) } )

Conversely, I made a ConvertTo-Markdown function which converts an object like that back into Markdown. It also handles the sorting of properties, methods, constructors and fields.

function ConvertTo-MarkDown { [CmdletBinding()] param ( [Parameter(Mandatory = $true, Position = 0, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true )] [ValidateNotNullOrEmpty()] [System.Collections.Generic.List[System.Collections.Hashtable]] $InputObject ) process { foreach ($Heading in $InputObject) { "# {0}" -f $Heading.Heading $Heading.Text $Subheadings = $Heading.Subheadings if ($Heading.Heading -match 'Properties|Methods|Fields') { $Subheadings = $Heading.Subheadings | Sort-Object {$_.Heading} } if ($Heading.Heading -match 'Constructors') { $Subheadings = $Heading.Subheadings | Sort-Object -Property @{ Expression = { if ($_.Heading -match '\(\)') {-1} else {($_.Heading -split ',').count} } Ascending = $true } } foreach ($Subheading in $Subheadings) { "## {0}" -f $Subheading.Heading $Subheading.Text } } } }

The Classy PlatyPS Puts It All Together

Being able to manage markdown as an object makes it easy to see what has and has not ben put in the documentation. We just need to compare "what was" with "what is" and generate "what ought". For that I created an Update-ClassMarkdown function. It's too lengthy to post here but you can see it in BuildDocs-Helper.ps1.

Finally, BuildDocs combines it all:

if ($ClassDocs) { "Processing Classes..." $Classes = Get-ModuleClass -ModuleName $ModuleName | Where-Object {$_.IsClass} $AboutHelpDocs = Get-ChildItem -Path $ModuleHelpPath -Filter 'about_*.md' foreach ($Class in $Classes) { $HelpDoc = $AboutHelpDocs | Where-Object {$_.basename -like "about_$($Class.Name)"} if ($HelpDoc) { Update-ClassMarkdown -Class $Class -Path $HelpDoc.FullName continue } $AboutPath = Join-Path $ModuleHelpPath "about_$($Class.Name).md" Classtext $Class | Set-Content -Path $AboutPath } }

There are similar patterns in place for Enums. They are only separated from classes so that text can use "Enum" instead of "Class" in the templates. Enum objects have Fields instead or properties and of course, those fields don't have any special data type. So the Enum code is pretty boring in comparison to classes, but it uses much of the same reflection techniques.

The Classy PlatyPS Scoffs at Privacy

PSRAW will be a community project. As such, we need good documentation for the Private Functions of the module just as much as we do for the Public Functions. That way contributors can hit the ground running much quicker without having to dig deeply into source to figure out the inner workings. PlatyPS does not include any out-of-the-box support for Module Private Functions. But, it will support automated updating of documentation for functions in general.

The Classy PlatyPS Demands to See Your Privates

The first problem to overcome is getting a list of private functions. Now, most projects, including mine, use a Private folder in the module root to store them. But, some projects may have private functions defined in other locations, including the .psm1. I wanted a universal way to get private functions from a module.  I didn't know where to start but then I realized that pester has this functionality to work with mocking functions. Digging around the source for Mock I discovered something really neat I did not know about.

If you store the result of Get-Module into a variable you can execute commands in the module's context like this:

$Module = Get-Module PSRAW & $Module {Get-Command -CommandType Function} | Where-Object {$_.Source -like 'PSRAW'}

The result will include all the functions the module sees for its own context. That includes both public and private functions! I created a Get-ModulePrivateFunction function to make this easy.

function Get-ModulePrivateFunction { [CmdletBinding()] [OutputType([System.Management.Automation.FunctionInfo])] param ( [Parameter( Mandatory = $true, ValueFromPipeline = $true, ValueFromPipelineByPropertyName = $true )] [string[]] $ModuleName ) process { foreach ($Name in $ModuleName) { $Module = $null Write-Verbose "Processing Module '$Name'" $Module = Get-Module -Name $Name -ErrorAction SilentlyContinue if (-not $Module) { Write-Error "Module '$Name' not found" continue } $ScriptBlock = { $ExecutionContext.InvokeCommand.GetCommands('*', 'Function', $true) } $PublicFunctions = $Module.ExportedCommands.GetEnumerator() | Select-Object -ExpandProperty Value | Select-Object -ExpandProperty Name & $Module $ScriptBlock | Where-Object {$_.Source -eq $Name -and $_.Name -notin $PublicFunctions} } } }

This means we can now use them in documentation and testing!!! I put both Get-ModuleClass and Get-ModulePrivateFunction in the ModuleData-Helper.ps1.

The Classy PlatyPS Documents Your Privates

Now that we have a way to get a list of private functions in a module, we need to make PlatyPS able to document them. Unfortunately, we cant just pass functions returned from Get-Command to PlatyPS and get documentation. The functions need to exist in memory. I had hoped that I could maybe call PlatyPS from within the module scope, but it didn't work. The simple solution is to load the definition results from Get-ModulePrivateFunction into memory and then call PlatyPS.

BuildDocs.ps1:

# ...Snip If ($PrivateDocs) { "Processing Private Functions..." $PrivateFunctions = Get-ModulePrivateFunction -ModuleName $ModuleName $PrivateHelp = Get-ChildItem $PrivateHelpPath -Filter '*.md' -ErrorAction SilentlyContinue foreach ($PrivateFunction in $PrivateFunctions) { $HelpDoc = $PrivateHelp | Where-Object {$_.basename -like $PrivateFunction.Name} $FunctionDefinition = "Function {0} {{ {1} }}" -f $PrivateFunction.name, $PrivateFunction.Definition . ([scriptblock]::Create($FunctionDefinition)) if (-not $HelpDoc) { $Params = @{ Command = $PrivateFunction.name Force = $true AlphabeticParamsOrder = $true OutputFolder = $PrivateHelpPath WarningAction = 'SilentlyContinue' } New-MarkdownHelp @Params } $Params = @{ Path = "$PrivateHelpPath\{0}.md" -f $PrivateFunction.Name AlphabeticParamsOrder = $true WarningAction = 'SilentlyContinue' } Update-MarkdownHelp @Params Remove-Item "function:\$($PrivateFunction.name)" -ErrorAction SilentlyContinue } } # Snip...

Why not just dot source the files? Well, how do we find what functions were exported without also doing an AST scan of the files? Also, dot sourcing the .psm1 should be troublesome depending on what all it does.We don't want to execute code so much as we want to create function definitions and get them loaded into memory.

The Classy PlatyPS Results

The BuildDocs script will create several files under the project's \docs folder. The \docs\Module folder contains the markdown documentation for Classes, Enums and Module Public Functions. The \docs\PrivateFunctions folder contains the markdown documentation for Module private functions

\DOCS │ ├───Module │ about_RedditApiResponse.md │ about_RedditApplication.md │ about_RedditApplicationType.md │ about_RedditOAuthCode.md │ about_RedditOAuthDuration.md │ about_RedditOAuthGrantType.md │ about_RedditOAuthResponseType.md │ about_RedditOAuthScope.md │ about_RedditOAuthToken.md │ Export-RedditApplication.md │ Export-RedditOAuthToken.md │ Get-RedditOAuthScope.md │ Import-RedditApplication.md │ Import-RedditOAuthToken.md │ Invoke-RedditRequest.md │ New-RedditApplication.md │ PSRAW.md │ Request-RedditOAuthToken.md │ Update-RedditOAuthToken.md │ └───PrivateFunctions Get-AuthorizationHeader.md Request-RedditOAuthCode.md Request-RedditOAuthTokenClient.md Request-RedditOAuthTokenCode.md Request-RedditOAuthTokenImplicit.md Request-RedditOAuthTokeninstalled.md Request-RedditOAuthTokenPassword.md Request-RedditOAuthTokenRefresh.md Show-RedditOAuthWindow.md Wait-RedditApiRateLimit.md

It will also generate the external help. I'm using the en-US culture. This is created under the \PSRAW\en-US folder.

\PSRAW\EN-US about_RedditApiResponse.help.txt about_RedditApplication.help.txt about_RedditApplicationType.help.txt about_RedditOAuthCode.help.txt about_RedditOAuthDuration.help.txt about_RedditOAuthGrantType.help.txt about_RedditOAuthResponseType.help.txt about_RedditOAuthScope.help.txt about_RedditOAuthToken.help.txt PSRAW-help.xml

The about_ topics each have their own .txt file and the Public Functions are in MAML format in the PSRAW-help.xml. Private Functions do not get added to the module's external help as they are not normally accessible from the console and therefore don't need to have in-console documentation.

Conclusion

This has been a blast to work on. Trying to figure this all out really expanded my understanding of PowerShell. It also exposed some painful shortcomings, both within PowerShell and within my own comprehension. In any event, I have been so excited to share this piece of my CI/CD pipeline. It's not something I'm seeing done any other projects.

It's still a bit immature. The sorting can definitely use some work. I skimped on it because I was rushing to get the release out. However, as it stands now, most of the auto-documentation of Classes, Enums, and Private Functions "just works"! It may not be perfect but it is functional beyond my initial expectations.

I only hope that this makes contributing to the PSRAW project easier. I have a full documentation requirement that will fail tests if any Function (public and private), Parameter, Class, Enum, Property, Method, Constructor, Field, or link is missing. The Classy PlatyPS should aid contributors by providing the scaffolding.  All they have to do is fill in the blanks!