Peanut Butter and Chocolate: Azure Functions CI/CD Pipeline with AWS CodeCommit (Part 2 of 6)

Part 2

In Part 1 I gave an overview of the Peanut Butter and Chocolate Project. In Part 2 I will cover the required PowerShell modules for the deployment, the settings used in the project, authenticating tp Azure and AWS, deploying the Azure Function App, and Deploying the AWS CodeCommit repository. I also demonstrate how to verify the resource deployments with Pester.

As a reminder, you can obtain the project code from https://github.com/markekraus/PeanutButterChocolate

Series Table of Contents

• Part 1 of 6: Project Introduction and Overview
  
• Part 2 of 6 (This Post): Project Settings, Authentication, Azure Function App Deployment, and AWS Code Commit Deployment
  
• Part 3 of 6: Creating AWS IAM User for Git, Generating HTTPS Git Credentials, Configuring Azure Web App External Git Deployment, and Initial Manual Deployment
• Part 4 of 6: Create AWS IAM Role, Deploy the C# .NET Core 2.0 AWS Lambda, and Create an AWS KMS Key
• Part 5 of 6: Add AWS CodeCommit Trigger and Encrypt Secrets with AWS KMS
• Part 6 of 6: Generate Configuration File, Automatic deployment from AWS CodeCommit to Azure Functions, Trigger the Azure Function from PowerShell, and Series Conclusion

Install and Import the Required Modules

The Configuration.ps1 starts off with installing and Importing the 3 modules used to deploy the pipeline: AzureRM, AWSPowerShell, and Pester. If you have the AzureRM and AWSPowerShell modules already installed, you may want to consider updating. I included the versions of the modules used at the time of authoring this project. Older versions may not work. Pester 4.2.0 was released 1 day before Part 1 of this series. I use features that were added in 4.2.0 in this project so you may very likely need to update Pester if you are following along with this series as it is being written.

# Install and Import AzureRM Module

Install-Module AzureRM -Scope CurrentUser -Force -MinimumVersion 5.2.0

Import-Module AzureRM -Force -MinimumVersion 5.2.0

# Install and Import the AWSPowerShell module

Install-Module -Name AWSPowerShell -Scope CurrentUser -Force -MinimumVersion 3.3.232.0

Import-Module -MinimumVersion 3.3.232.0 -Name AWSPowerShell

# Install and Import Pester 4.2.0

# This script makes use of features only available in Pester 4.2.0 and up

Install-Module Pester -Scope CurrentUser -Force -MinimumVersion 4.2.0

Import-Module Pester -Force -MinimumVersion 4.2.0

Settings

Before we can begin adding resource, we need to define all the settings for the project. I use a $Settings HashTable for this. Using a settings HashTable makes it easier to find all the places in the script where the settings are used. It also makes intellisense easier to use in the project while developing it. Finally, this also make it easy to move the settings out to a psd1 file and then import it with Import-PowerShellDataFile without having to make significant changes to the logic in the code.

# Prompt for the AWS Admin access Key and Secret Key

$AWSCredentials = Get-Credential -Message 'AWS Access Key and Secret Key'

# set the Base Directory for the project

$BaseDir = $PWD

if ($PSScriptRoot) {

    $BaseDir = $PSScriptRoot

}

First thing we do is grab the AWS Access Key and Secret Key for the AWS admin user. The AWS cmdlets actually take these both as plain-text strings. Later in the settings HashTable I convert the Secret Key from a SecureString to a String. I'm not fond of secrets ever being in plain-text, so I often use this method to keep plain-text secrets safe when the cmdlet doesn't accept SecureStrings for secrets.

Next, we establish the base directory. You should run the Configuration.ps1 from the root directory of the project. I use this method because I often run scripts from VS Code with highlighting sections and pressing F8 to execute one chunk of code at a time. The $PSScriptRoot variable is not populated unless you are actually invoking a script. In which case, $PWD is used to establish the base directory in an console or editor session. The $BaseDir variable is used by several settings to determine locations for project source files.

I included comments for each of the settings explaining what they are. Rather list them off one-by-one here, I will just post the settings HashTable and you can refer to the comments within.

# Settlings used by this project

$Settings = @{

    SrcDirectory                  = Join-Path $BaseDir 'src'

    # Folder under which the local Git repository will be cloned

    GitDirectory                  = 'c:\Git'

    # URL to the Azure Resource Template used to deploy the Azure Function Web App

    ResourceTemplateUrl            =

        'https://raw.githubusercontent.com/Azure/azure-quickstart-templates/57f091bc3c7d298e102ab092a1a25399b49d77f3/101-function-app-create-dynamic/azuredeploy.json'

    # New Azure Resource Group Name under which to deploy the Azure Web App and Storage

    ResourceGroupName             = 'PBnC'

    # The name to use for the Azure Function Web App

    AppName                       = 'PBnC'

    # See the -Location parameter of New-AzureRmResourceGroup for details

    ResourceGroupLocation         = "South Central US"

    # Options: Standard_LRS, Standard_GRS, Standard_RAGRS

    FunctionAppStorageAccountType = 'Standard_LRS'

    # The AWS Access Key and Secret key of the admin account used to configure AWS resources

    AwsAccessKey                  = $AWSCredentials.UserName

    AWSSecretKey                  = $AWSCredentials.GetNetworkCredential().Password

    # Name of an AWS credential profile to create and use to configure AWS resources

    AwsProfile                    = 'Aws01'

    # The AWS region in which to configure resources

    AwsRegion                     = 'us-east-2'

    # The Name to use for the CodeCommit Repository

    CCRepositoryName              = 'PBnC'

    # The Description to set on the CodeCommit Repository

    CCRepositoryDescription       = 'Peanut Butter and Chocolate'

    # Username of an IAM user to create.

    # This user will granted pull access to the CodeCommit repository

    # Its HTTPS Git credentials for AWS CodeCommit will be used in azure to pull from the repository

    CCGitUser                     = 'PBnC-Git-User'

    # Name to use for the inline policy applied to the CCGitUser to allow pull access to the CodeCommit repository

    CCGitUserPolicyName           = 'Allow-GitPull-to-CodeCommit-PBnC'

    # Name to use for the CodeCommit repository trigger used to invoke the lambda function

    CCTriggerName                 = 'TriggerAzureFunctionDeployment'

    # Statement ID (SID) of the lambda policy to allow the CodeCommit repository to invoke the lambda

    CCLambdaPolicyStatementId     = 'CodeCommit-PBnC-Invoke-TriggerAzureFunctionDeployment'

    # Name of the AWS C# Lambda Function that CodeCommit will invoke to trigger Azure Web App Deployment

    LambdaName                    = 'TriggerAzureFunctionDeployment'

    # The Handler definition used by the C# Lambda Function

    LambdaHandler                 = 'PBnCLambda::PBnCLambda.Function::FunctionHandler'

    # The Runtime used by the AWS C# Lambda

    LambdaRuntime                 = 'dotnetcore2.0'

    # The description to set for the AWS C# Lambda

    LambdaDescription             = 'Triggers a manual deployment of an Azure Web App from CodeCommit.'

    # The name to give the IAM Role the AWS Lambda Function will assume when it is invoked

    LambdaRoleName                = 'TriggerAzureFunctionDeploymentRole'

    # The description to give the IAM Role the AWS Lambda Function will assume when it is invoked

    LambdaRoleDescription         = 'Role assumed by the TriggerAzureFunctionDeployment Lambda'

    # Manged IAM policies to apply to the IAM Role

    LambdaRolePolicyArns          = @(

        'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'

        'arn:aws:iam::aws:policy/AWSCodeCommitReadOnly'

    )

    # Directory containing the C# .NET Core project for the AWS Lambda Function

    LambdaSrcDirectory            = Join-Path $BaseDir 'CSLambda\PBnCLambda\'

    # In testing, ths Lambda takes 2-5 seconds to run and consumes about 90MB of RAM

    # Timeout in seconds to set for the AWS Lambda Function executions

    LambdaTimeOut                 = 60

    # The memory size to use for the AWS Lambda Function

    LambdaMemorySize              = 128

    # Description to use for the AWS KMS key that will be generated

    # This key will be used by user to encrypt string in the cc2af.yml file

    # It will also be used by the AWS Lambda Function to decrypt those same strings

    KMSKeyDescription             = 'Key Used by TriggerAzureFunctionDeployment to manage config secrets.'

    # Name to set for the inline policy applied to the Lambda Role granting it access to the KMS Key

    KMSRoleAccessPolicyName       = 'Allow-TriggerAzureFunctionDeployment-Encrypt-Decrypt'

}

The AppName setting will need to be something different. That must be unique within Azure.

The ResrouceGroupLocation and AwsRegion settings need to support Azure Function Apps and AWS CodeCommit respectively.

Asset HashTables

I use 2 hashtables to store results from all the resource related commands. One HashTable is used for all Azure results and one HashTable is used for all AWS results. Several commands rely on information produced by the results of other commands. Those will also go in the asset HashTables. This is done to keep results grouped together and to assist with intellisense. 

# Hashtables used for storing results from commands

$AzureAssets = @{}

$AWSAssets = @{}

Authenticating to Azure and AWS

Next we get our our authentication configured. For Azure we actually authenticate and create a login session. For AWS, we are merely creating a credential profile and setting it as default. AWS cmdlets perform automatic authentication as needed, so this doesn't truly log you in or start an authentication session with AWS. If you messed up your AWS Access Key or Secret Key, you wont know until the first AWS commands are run.

# Log in to Azure

$AzureAssets['AccountLogin'] = Login-AzureRmAccount

# Create AWS credential profile and set it as the default configuration

Set-AWSCredential -StoreAs $Settings.AwsProfile -AccessKey $Settings.AwsAccessKey -SecretKey $Settings.AWSSecretKey

Initialize-AWSDefaultConfiguration -ProfileName $Settings.AwsProfile -Region $Settings.AwsRegion

I left Azure using the interactive login approach. This is a demo and since there is already one part that must be done interactively, I decided against including unattended login. I also did this out of fear of accidentally including my credentials in the demo. If you want to use certificate authentication or other means, you are welcome to update the code here before you run it.

Create the Azure Resource Group and Deploy an Azure Functions Web App

With all of the requirements out of the way, we can finally begin deploying resources. The first resource to deploy is the Azure Resource Group. This is a logical container that will contain the Azure Function Web App and its storage account. This will be deployed in the azure location specified in the settings. You must use a location where Azure Function Apps are supported.

# Create the Resource Group and add a Function App Deployment

$Params = @{

    Name     = $Settings.ResourceGroupName

    Location = $Settings.ResourceGroupLocation

}

$AzureAssets['ResourceGroup'] = New-AzureRmResourceGroup @Params

With the resource group created, we can now deploy resources to the resource group. I wanted to try out template based deployment so I chose that method for this project. I am unsure if Azure Function Apps can be deployed without a template, as I haven't tried. I have deployed Web Apps with PowerShell cmdlets before, but never Function Apps. This deployment template is the base template provided by Microsoft https://github.com/Azure/azure-quickstart-templates/tree/master/101-function-app-create-dynamic. It will deploy a consumption plan Function App and its storage account. The storage account options are listed in the settings.

# This Template deploys the storage account, App Service account, and Function App.

$Params = @{

    TemplateUri        = $Settings.ResourceTemplateUrl

    Name               = $Settings.ResourceGroupName

    ResourceGroupName  = $Settings.ResourceGroupName

    appName            = $Settings.AppName

    storageAccountType = $Settings.FunctionAppStorageAccountType

}

$AzureAssets['Deployment'] = New-AzureRmResourceGroupDeployment @Params -ErrorVariable 'DeploymentErrors'

Now we use Pester to verify the deployment:

# Validate that the deployment was successful

Describe "Deployment of '$($Settings.ResourceGroupName)' Azure Function App" {

    It "Was Successful." {

        $DeploymentErrors | Should -HaveCount 0

        $AzureAssets.Deployment.ProvisioningState | Should -BeExactly 'Succeeded'

    }

    It "Has a Storage Account." {

        $AzureAssets['Storage'] = Get-AzureRmStorageAccount -ResourceGroupName $Settings.ResourceGroupName

        $AzureAssets.Storage.ProvisioningState | Should -BeExactly 'Succeeded'

    }

    It "Has an App Service Plan." {

        $AzureAssets['AppService'] = Get-AzureRmAppServicePlan -ResourceGroupName $Settings.ResourceGroupName

        $AzureAssets.AppService.Status | Should -BeExactly 'Ready'

        $AzureAssets.AppService.NumberOfSites | Should -Be 1

    }

    It "Has a Web App." {

        $AzureAssets['WebbApps'] = Get-AzureRmWebApp -ResourceGroupName $Settings.ResourceGroupName

        $AzureAssets.WebbApps | Should -HaveCount 1

        $AzureAssets.WebbApps[0].State | Should -BeExactly 'Running'

        $AzureAssets.WebbApps[0].SiteName | Should -Be $Settings.AppName

    }

}

If all goes well you should see results like this:

In the azure portal the Resource Group should have an App Service Plan, and App Service and Storage Account

In the Function Apps the Function App should be visible.

There won't be any functions visible yet as we have not deployed any.

Get The Azure Function Web App Deployment Credentials

With the Function App deployed we can now obtain the deployment credentials.

# Grab the PublishProfile for the Web App. This gives us the deployment username and password

$AzureAssets['WebAppPublishingProfile'] = [xml](Get-AzureRmWebAppPublishingProfile -WebApp $AzureAssets.WebbApps[0])

$AzureAssets['WebAppUserName'] = $AzureAssets.WebAppPublishingProfile.publishData.publishProfile[0].userName

$AzureAssets['WebAppUserPwd'] = $AzureAssets.WebAppPublishingProfile.publishData.publishProfile[0].userPWD

$AzureAssets['WebAppDeployUrl'] = 'https://{0}.scm.azurewebsites.net:443/deploy' -f

    $AzureAssets.ResourceGroup.AppName

# Create the Azure Web App Kudu Authorization header

# This is used to retrieve the Master key and Function key required to invoke the Azure Function

$AzureAssets['KuduAuth'] = 'Basic {0}' -f [Convert]::ToBase64String(

    [System.Text.Encoding]::UTF8.GetBytes(

        ('{0}:{1}' -f $AzureAssets.WebAppUserName, $AzureAssets.WebAppUserPwd)

    )

)

The deployment credentials will later be included in a YAML file stored in the AWS CodeCommit repository with the password encrypted by AWS KMS. These credentials are used to trigger an external git deployment by posting a JSON submission to the Deployment Trigger URL.

KuduAuth is a basic authorization HTTP request header that will be used later to obtain the the master key and function key. These keys are used to allow the Azure Function to be triggered via HTTP. Kudu is engine that Azure Web Apps use for git deployments. It also controls the keys used for HTTP triggers. It you are trying to figure out automation for Web Apps and Functions, you will want to familiarize yourself with Kudu's API.

Deploy the AWS CodeCommit Repository

The AWS CodeCommit repository is where all the code for the Azure Functions will live for this particular Function App. If deploying multiple function apps, You will need to deploy one repository per Function App. The file and folder structure of that repository is rigid if you use the manual deployment method (which is what I use in this project). I will go in more detail about that structure later, but the key here is that using this POC requires one AWS CodeCommit repository per Azure Function App. You would need to devise a more custom deployment to house multiple Azure Functions Apps in a single CodeCommit.

The creation of the CodeCommit repository is very simple. It just requires a name and a description. It will be deployed in the AWS region set in the settings or the current region configured for your defaults This will be true of all AWS resources deployed. CodeCommit is the most restrictive as to its available regions. So make sure you select a region in the settings that can support CodeCommit.

# Create the AWS CodeCommit Repository

$Params = @{

    RepositoryName        = $Settings.CCRepositoryName

    RepositoryDescription = $Settings.CCRepositoryDescription

}

$AWSAssets['CCRepository'] = New-CCRepository @Params -ErrorVariable 'CodeCommitRepositoryErrors'

Now we validate the CodeCommit deployment with Pester.

# Validate the CodeCommit Repository was created

Describe "Deployment of '$($Settings.CCRepositoryName)' AWS CodeCommit Repository" {

    It "Was Successful." {

        $CCRepository = Get-CCRepository -RepositoryName $Settings.CCRepositoryName

        $CCRepository.RepositoryName | Should -BeExactly $Settings.CCRepositoryName

        $CCRepository.RepositoryDescription | Should -BeExactly $Settings.CCRepositoryDescription

    }

}

If it is successful you should see this:

It should also be visible in the AWS Management Console:

Part 2 End

I'm trying not to make monolithic posts and to break this series up into smaller parts that get published quickly. In Part 3 we will continue with creating more resources. I'm going to try and publish each part less than a week apart. This may be interrupted by the MVP Summit the first week of March.