Multi-Account Deployment Using GitHub Actions and the AWS Cloud Development Kit (CDK)

Deploying resources across multiple AWS accounts can be a challenging task. This blog post outlines Big Cloud Country’s approach to streamline this process using Git flow, trunk-based development, GitHub Actions, and the AWS Cloud Development Kit (CDK).

Development Workflow

Our team adopted an advanced and structured development workflow that combined elements of Git flow and trunk-based development practices. This workflow was designed to streamline the development process, ensure code quality, and facilitate continuous integration and deployment. Here's a step-by-step breakdown of our flow:

1. Pulling the Latest Code: A developer begins by pulling the latest code from the develop branch. This ensures that they are working with the most up-to-date version of the codebase, minimizing conflicts and discrepancies.

2. Creating a Feature Branch: From the develop branch, the developer creates a new feature branch. This branch is dedicated to a specific task or feature, allowing for focused and isolated development work.

3. Pushing the Feature Branch: Once the initial work on the feature is completed, the developer pushes the feature branch to the repository. This makes the branch available for collaboration and further development by other team members.

4. Opening a Pull Request to Develop Branch: The developer then opens a pull request (PR) from the feature branch to the develop branch. This PR serves as a request to merge the new code into the development branch.

5. PR Testing and Review: The pull request triggers automated tests to ensure code quality and functionality. Simultaneously, it undergoes a thorough review process by other team members or designated reviewers. This step is crucial for maintaining code quality and catching potential issues early.

6. Merging PR to Develop Branch: Once the PR passes all tests and reviews, it is merged into the develop branch. This merge signifies that the feature is ready for further testing in the development environment.

7. Development Testing: The develop branch, now containing the new features, is tested in the development environment. This step allows developers to identify and fix any issues that might arise from the integration of new code.

8. Opening a PR to Main Branch: After successful testing in the development environment, the developer opens a pull request from the develop branch to the main branch. This PR is intended to merge the tested features into the main codebase.

9. Main Branch PR Testing and Review: Similar to the earlier PR, this pull request also undergoes a series of automated tests and code reviews. The focus here is on ensuring that the new features are fully compatible with the existing codebase and are ready for deployment to production.

10. Merging PR to Main: Once the final PR is approved and passes all tests, it is merged into the main branch. This merge marks the completion of the feature's development cycle and its readiness for deployment to the production environment.

By following this meticulous development workflow, our team ensured that each feature was developed, tested, and reviewed thoroughly before being integrated into the main codebase. This process not only maintained high standards of code quality but also facilitated smooth and continuous delivery of new features and updates.

Enforcing Branch Protection

To ensure the integrity and stability of our codebase, we implemented strict branch protection rules for both main and develop branches in our repository. These rules are designed to prevent direct pushes to these branches, requiring changes to be made through a more controlled and reviewable process of pull requests (PRs).

No Direct Pushes to Main or Develop

• Main Branch: The main branch, being the primary source of truth for our production environment, is tightly controlled. Direct pushes to main are disallowed. Changes must be introduced through a pull request from the develop branch.

• Develop Branch: Similarly, direct pushes to the develop branch are prohibited. This branch acts as an integration point for new features and bug fixes. Developers must create feature branches off develop, and upon completion, raise a PR to merge their changes back into develop.

GitHub Actions Integration

We leveraged GitHub Actions to create an efficient CI/CD pipeline. This included reusable workflows, GitHub environments, and matrix strategies for simultaneous deployments to multiple AWS accounts. Key elements of our GitHub Actions setup include:

Leveraging Reusable Workflows in GitHub Actions

What is a Reusable Workflow?

In GitHub Actions, a reusable workflow is a modular and independent sequence of tasks that can be invoked or ‘called’ from other workflows within the same repository or across different repositories. This concept promotes reusability, maintainability, and scalability in CI/CD processes. Instead of duplicating code across multiple workflow files, a reusable workflow allows teams to define a set of tasks once and use them in various contexts.

Advantages of Reusable Workflows

• Efficiency: Reduces repetition and redundancy in workflow definitions.

• Consistency: Ensures uniformity in the execution of common tasks across projects.

• Ease of Maintenance: Updates to a reusable workflow propagate to all workflows that utilize it, simplifying maintenance.

Our Deploy Workflow: A Deep Dive

In our CI/CD pipeline, the Deploy workflow is designed as a reusable workflow. It’s specifically crafted to handle the deployment of our AWS CDK resources. This workflow can be triggered by other workflows, such as Deploy Dev or Deploy Main, which are tailored for development and main branch deployments, respectively.

Workflow Activation and Input

The following YAML code snippet illustrates how this reusable workflow is structured:

on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
        description: 'The GitHub environment to deploy against.'

• Trigger: The workflow is triggered by workflow_call. This trigger is key in making a workflow reusable, as it enables other workflows to call this workflow.

• Inputs: We define an input parameter environment, which is essential for the deployment process. This parameter specifies the target GitHub environment for the deployment.

• required: true indicates that this input must be provided when the workflow is called.

• type: string specifies that the input value should be a string.

• description: provides a clear explanation of what the environment input represents, enhancing clarity and usability for users invoking this workflow.

Harnessing the Power of Matrix Strategy in GitHub Actions

Understanding the Matrix Strategy in GitHub Actions

Matrix strategy in GitHub Actions is a powerful feature designed to execute workflows across multiple configurations. This strategy allows you to run a workflow in parallel across different combinations of environment variables, operating systems, programming languages, or any other configurable parameters. It’s particularly useful for testing across multiple environments or deploying to different targets. For example, this allowed us to concurrently deploy our data analysis pipeline stack to our four different production accounts.

Key Benefits of Using Matrix Strategy

• Parallel Execution: Speeds up the execution time as jobs run simultaneously across different configurations.

• Flexibility: Easily test or deploy across various configurations without duplicating workflow definitions.

• Scalability: Add more configurations as needed by simply expanding the matrix.

Application in Deploy to Environment Workflow

In our CI/CD pipeline, the Deploy to Environment workflow effectively leverages the matrix strategy for deployment purposes. This strategy is crucial for deploying to multiple AWS accounts or environments in a consistent and efficient manner.

Workflow Snippet Explanation

Consider the following code snippet from our Deploy to Environment workflow:

jobs:
  deploy:
    strategy:
      fail-fast: false
      matrix:
        environment: [account1, account2, account3, account4]
    name: Deploy waves to ${{ matrix.environment }}
    uses: ./.github/workflows/deploy.yaml
    with:
      environment: ${{ matrix.environment }}

• Matrix Declaration: matrix: environment: [account1, account2, account3, account4] defines a matrix of environments. Here, environment is a variable that takes on the values account1, account2, account3, and account4 in different instances of the workflow run.

• Parallel Deployments: For each value in the matrix, a separate instance of the workflow is run in parallel. This means that four jobs will be executed concurrently, each deploying to a different AWS account or environment as specified in the matrix.

• Dynamic Job Names: name: Deploy waves to ${{ matrix.environment }} dynamically names each job based on the matrix value, enhancing readability and tracking in the CI/CD process.

  • Workflow Reusability: The line uses: ./.github/workflows/deploy.yaml calls the reusable Deploy workflow, passing the current matrix.environment value as an input. This showcases how different components of GitHub Actions can be integrated for sophisticated workflow designs.

Fail-fast Strategy

The fail-fast: false setting is noteworthy. By default, when one job in a matrix fails, all other jobs are cancelled. Setting fail-fast to false ensures that all jobs in the matrix run to completion, regardless of the success or failure of others. This is important for deployment scenarios where the success of one environment should not depend on others.

By incorporating the matrix strategy in our Deploy to Environment workflow, we’ve created a robust mechanism to manage deployments across multiple environments simultaneously. This approach is not only efficient but also aligns with best practices for scalable and maintainable CI/CD pipelines.

Enhancing Deployment with GitHub Environments

Understanding GitHub Environments

GitHub Environments are a feature within GitHub Actions that enable more controlled deployments by managing environment-specific configurations and secrets. Environments can be defined at the repository level and are used to group environment-specific settings, like secrets, deployment branches, or review policies.

Key Features of GitHub Environments

• Environment-specific Secrets: Store and access secrets that are unique to each deployment environment.

• Deployment Branch Rules: Set specific branches that can trigger deployments to certain environments.

• Review Policies: Implement required reviews or manual approvals for deployments.

Integration with CI/CD Workflow

In our CI/CD pipeline, GitHub environments play a crucial role in aligning our deployment process with the configuration specified in our cdk.context.json file.

Aligning with cdk.context.json

The cdk.context.json file is used in AWS CDK applications to define context values that can vary between different deployment environments. For example, it can contain different AWS account IDs, feature flags, or region settings for development, staging, and production environments.

Workflow Configuration Example

Here’s an example of how we align GitHub environments with cdk.context.json:

jobs:
  deploy:
    name: Deploy to ${{ inputs.environment }} 
    environment: ${{ inputs.environment }}
    env:
      ENVIRONMENT: ${{ vars.ENVIRONMENT }}
      CDK_DEPLOY_ACCOUNT: ${{ vars.ACCOUNT_ID }}
      CDK_DEPLOY_REGION: ${{ vars.REGION }}

• Setting Environment Variables: In this configuration, the env section sets environment variables on the GitHub runner. These variables (ENVIRONMENT, CDK_DEPLOY_ACCOUNT, and CDK_DEPLOY_REGION) are crucial for the deployment process and are derived from the GitHub environment’s settings.

• Dynamic Environment Selection: The environment: ${{ inputs.environment }} line dynamically selects the GitHub environment based on the input provided to the workflow. This approach allows us to have a single workflow configuration that can adapt to multiple deployment contexts.

Benefits in Deployment

By integrating GitHub environments with our cdk.context.json file, we gain several benefits:

• Consistency: Ensures that the deployment process is consistently using the correct settings for each environment.

• Security: Allows sensitive information like AWS account IDs to be stored securely in GitHub environments.

• Flexibility: Makes it easy to update environment configurations without changing the workflow files.

Application Deployment

In our deployment strategy, the application’s environment configuration plays a pivotal role. Below is a snippet from our app.py file, demonstrating how we use the environment variable set on the GitHub runner to determine the target deployment environment. We then use the AWS CDK function try_get_context to fetch environment-specific values defined in our cdk.context.json file:

# CUSTOMER NAME OBTAINED FROM GITHUB ACTION ENV VARIABLE
customer = os.environ.get("ENVIRONMENT")
# ENVIRONMENT CONTEXT VALUES OBTAINED FROM cdk.context.json
environment_config = app.node.try_get_context(customer)

An example of the cdk.context.json file, which defines environment-specific values, is shown below:

{
  "example_environment": {
    "feature_flag": true,    
    "account_id": "123456789101",    
    "region": "us-east-2",    
    "project": "example_project",    
    "environment": "example_environment"  
   }
}

Mapping to cdk.context.json: The values set in GitHub environments correspond to the keys in the cdk.context.json file. This ensures that the deployment process uses the correct configuration for each environment.

AWS Credentials Configuration

In our project, we implemented a sophisticated strategy for managing deployment access and integrating our GitHub repository with our AWS accounts. This strategy hinged on a centralized deployment model. Within this model, we used CloudFormation StackSets in our administrative account to establish IAM deployment roles and facilitate an OpenID Connect (OIDC) connection between each AWS account and our GitHub repository. This approach not only streamlined our deployment processes but also ensured secure and efficient integration across platforms. 

# Configure AWS Creds
- name: Configure AWS Credentials ${{ inputs.environment }}
  uses: aws-actions/configure-aws-credentials@v2
  with:
    role-to-assume: arn:aws:iam::${{ vars.ACCOUNT_ID }}:role/${{ vars.DEPLOYMENT_ROLE}}
    role-session-name: cdk-deployment-${{ vars.REGION }}-${{ vars.ACCOUNT_ID }}
    aws-region: ${{ vars.REGION }}

Diff Workflow

Similar to our deployment strategy, the Diff workflow allows us to preview expected changes before deployment. It’s configured to run on pull requests opened to our develop and main branches.

Conclusion

By integrating Git flow, GitHub Actions, and AWS CDK, we’ve created a robust and scalable multi-account deployment strategy. This approach not only streamlines the deployment process but also ensures consistency and reliability across different environments.