Making Modular CloudFormation with Includes

By Thomas Vachon

Pushing the CloudFormation Bleeding Edge: Native Modular Templates

When the YAML format for CloudFormation was launched in September 2016, many of the users knew it was only a matter of time until the commonly used pattern of including multiple YAML files into a single file made its way into CloudFormation. On March 28, 2017, AWS did exactly that by launching the AWS::Include Transform, albeit with surprising lack of fanfare.

While YAML was not a prerequisite to having this feature, it made it infinitely easier leverage as an end-user. There are very important things which I have discovered as I integrated AWS::Include into my daily work; some of these are documented fully, others are documented partially, and others not at all.

Terms of Use:

  • Partials - The snippets of CloudFormation stored in S3
  • Master - The template executed by the end-user
  • Includes - The AWS::Includes Transform

Key Points

  • Your Partials may be in either JSON or YAML
  • Partials must use only the long form of a Function Call (e.g Fn::Sub not !Sub)
  • Change sets are required for use of Includes
  • Partials must be accessible by the end-user’s STS assumption for CloudFormation
    • Public ACL Read is not required if you have a good bucket policy
  • Partials are included into the Master before evaluation of functions
    • Prevents using Fn::Sub in a Location directive for dev/prod s3 path of the Partials
  • Errors which occur in Partials create unusual errors on evaluation
  • Understanding scope is very important
  • Nested Includes calls is not supported (i.e. your partials template cannot have another partial in it)

Now I will dive into some of these key points in detail, use cases for where to AWS::Include, and lessons learned from living on the bleeding edge.

In a future post, I will detail use cases where Includes is ideal for your business such as creating predictable IAM Roles or a multi-engine RDS template

Key Points in Details

Execution Model

The execution model when using Includes is through CloudFormation Change Sets, which is a great way to enforce a known checkpoint but brings in difficulties for people who don’t use CloudFormation daily. When you use a Includes and you want to make a new stack, you are left with two options:

  1. Create the stack within the AWS Console - the console automatically creates a blank stack, change set, and prompts for CAPABILITY_NAMED_IAM
  2. Create a “blank” stack (e.g. just a wait handle) and then create a change set against that stack with CAPABILITY_NAMED_IAM in the create-change-set call
Mixing JSON and YAML

This adventure to using Includes was my first significant effort into using YAML for all my CloudFormation, up until this point the need for YAML (specifically inline comments) was not worth the time it would take to rewrite what we had been using to date.

One of the use-cases I leveraged Includes for is deploying IAM Policies and Roles for Federated Login as it requires predictable Role naming. I have found in practice it is easier to write the policies in JSON but as I was doing YAML now, I decided to keep it in pure YAML for readability.

AWS has, thankfully, provided the ability to continue keeping your Partials in either language regardless of the Master’s language. You may choose to keep your JSON templates for things like IAM Policies and use YAML for the Master.

What I did leverage from time to time was cfn-flip to ensure my YAML syntax was inline with the JSON evaluation. As the templates are included they are converted to JSON, so doing this is a reasonable checkpoint for yourself.

Evaluation Logic and Order

As I said above you cannot have any other Function evaluated before the includes happens. This means that you cannot do something like this:

Parameters:
  PartialsEnv:
    Type: String
    Default: prod
    AllowedValues:
      - prod
      - dev
...

Resources:
  MyTestResource:
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location:
          Fn::Sub: s3://my-partials-bucket/${PartialsEnv}/resources/test_resource.yaml

This will throw an error that you must provide a valid S3 URI/Object. I have raised this with support and an RFE has been created to allow this or something like it be accepted.

While the order of operations is not specific to Includes, I was bitten by the unwritten order more than once. For your reference, I have compiled an incomplete list of the evaluation precedence steps here:

  1. Mappings
  2. References Lookups
  3. Conditional Statements
  4. Substitutions

I will try to obtain the information on a complete list of these in the future and make a separate post on that.

The reason I included these is that you are more likely to run into trying to do Mappings inside Conditionals using Reference Lookups and that will fail with unpredictable results.

Errors & Debugging

I have compiled a list of the most common issues I have run into within Includes and YAML in general

Common Errors Causes
Circular Dependancies Incorrect/Invalid Reference in the Partial
Invalid Policy Syntax for IAM on execution A * IAM Policy Resource was not "*"
Invalid MajorEngineVersion ##.# for SqlServer Option Group SqlServer Option Groups are ##.## and during the YAML to JSON conversion, it drops the superfluous 4th digit, quoting such as "13.00" fixes this
Must Provide valid S3 URI or S3 Object 1. You are referencing a private S3 Object with no Bucket Policy
  2. You are trying to do a Function within the S3 Location call
  3. You are trying to reference a S3 Object which has no valid CloudFormation items

Scope and Use In Practice

When using Includes, its very important to pay attention to how Scope is in use. You cannot have two Includes calls in a single Scope level. I have detailed interesting use cases around scope below and its important all of these items are mutually exclusive within a single template (#1) or within sections (#2/#3).

1 - Use the transform section of the template

In this example I show how you can replace an entire template with Includes except Parameters, which can not be part of Includes.

Master Template

---
  AWSTemplateFormatVersion: 2010-09-09
  Transform:
    Name: AWS::Include
      Parameters:
        Location: s3://my-partials-bucket/my_stack.yaml

  # Parameters cannot be in an Includes
  Parameters:
    MyParam:
      Type: String

Partials Template

Metadata:
  ... # Your Metadata
Conditionals:
  ... # Your Conditionals
Mappings:
  ... # Your Mappings
Resources:
  ... # Your Resources
Outputs:
  ... # Your Outputs
2 - A section level

In this example I show how you can take all of the Resources, Outputs, etc of a template and put them into a Partial

Master Template

Mappings:
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: s3://my-partials-bucket/mappings/my_mappings.yaml

# Parameters cannot be in an Includes
Parameters:
  MyParam:
    Type: String

...

Resources:
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: s3://my-partials-bucket/resources/my_resources.yaml

Outputs:
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: s3://my-partials-bucket/outputs/my_outputs.yaml

Partials Templates

# my_mappings.yaml
AWS::CloudFormation::Interface:
 ParameterGroups:
  -
    Label:
      default: Global Account Information
    Parameters:
      - MyParam
 ParameterLabels:
    # These values must be quoted to add white space
    MyParam:
      default: 'My Parameter: '
# my_resources.yaml
MyLogicalKMSResourceName:
  Type: AWS::KMS::Key
  Properties:
    Description: |
      My KMS Example Resource
    Enabled: true
    ...

MyLogicalWaitResourceName:
  Type: AWS::CloudFormation::WaitConditionHandle
# my_outputs.yaml
MyLogicalKMSResourceOutput:
  Description: |
    KMS ARN Example
  Value:
    Ref: MyLogicalKMSResourceName
  Export:
    Name:
      Fn::Sub: ${AWS::StackName}-MyLogicalKMSResourceOutput
3 - Multiple Resources

In this example I show you can use excludes to abstract the body of each resource and output into their own Partial

...
Parameters:
  MyParam:
    Type: String

...

Resources:
  MyLogicalKMSResourceName:
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location: s3://my-partials-bucket/resources/kms_key.yaml
  MyLogicalWaitResourceName:
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location: s3://my-partials-bucket/resources/wait_handle.yaml

...

Outputs:
  MyLogicalKMSResourceOutput:
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location: s3://my-partials-bucket/outputs/kms_key.yaml
...

Partials Templates

#resources/kms_key.yaml
Type: AWS::KMS::Key
Properties:
  Description: |
    My KMS Example Resource
  Enabled: true
  ...
#resources/kms_key.yaml
Type: AWS::CloudFormation::WaitConditionHandle
#outputs/kms_key.yaml
Description: |
  KMS ARN Example
Value:
  Ref: MyLogicalKMSResourceName
Export:
  Name:
    Fn::Sub: ${AWS::StackName}-MyLogicalKMSResourceOutput
4 - Within a resource

This example shows how to use Includes to provide some modularity to the end-user while maintaining some attributes which are common in the Partial. When you get to this method, as the warning below notates scope is very tricky and you should take care. In the example I create an RDSDBParameterGroup which allows the Master Template to specify what the Parameters are in use for this RDS. Additionally, I show a second method of advanced scoping which allows you to have an Includes at the end of a section which provides any “generic” items, as a result of scoping, any use of this must be at the end of the Section or it will be overridden. I also demonstrate that regardless of how a resource is declared (e.g. RDSDBParameterGroup is “within a resource”) the Logical ID persists in the template after compilation (e.g the Output for RDSDBParameterGroup is in the Generic Outputs Includes)

Note: This method is considered advanced and requires significant testing

Master Template

Mappings:
...

# Parameters cannot be in an Includes
Parameters:
MyParam:
  Type: String

...

Resources:
  RDSDBParameterGroup:
    Type: AWS::RDS::DBParameterGroup
      Properties:
        # Common items for all Parameter Groups
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: s3://my-partials-bucket/resources/rds_parameter_group.yaml
        # Custom Parameters per RDS which are set by the "stack owner"
        Parameters:
          sql_mode: IGNORE_SPACE
          timezone: UTC

  # Must be the last item in the section
  # Includes a series of generic resources
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: s3://my-partials-bucket/resources/general_resouces.yaml

Outputs:
  MyLogicalKMSResourceOutput:
    Fn::Transform:
      Name: AWS::Include
      Parameters:
        Location: s3://my-partials-bucket/outputs/kms_key.yaml

  # Must be the last item in the section
  # Includes a series of generic outputs
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location: s3://my-partials-bucket/outputs/general_outputs.yaml

Partials Templates

# resources/rds_parameter_group.yaml
Description: RDS DB parameter group
Family: rdsengine-12.3
Tags:
-
  Key: Name
  Value: rds-engine-12.3-parameter-group
# resources/general_resouces.yaml
MyLogicalKMSResourceName:
  Type: AWS::KMS::Key
  Properties:
    Description: |
      My KMS Example Resource
    Enabled: true
    ...

MyLogicalWaitResourceName:
  Type: AWS::CloudFormation::WaitConditionHandle
# outputs/kms_key.yaml
Description: |
  KMS ARN Example
Value:
  Ref: MyLogicalKMSResourceName
Export:
  Name:
    Fn::Sub: ${AWS::StackName}-MyLogicalKMSResourceOutput
# outputs/general_outputs.yaml
RDSDBParameterGroupOutput:
  Description: |
    A RDS Parameter Group Example
    Value:
      Ref: RDSDBParameterGroup
    Export:
      Name:
        Fn::Sub: ${AWS::StackName}-RDSDBParameterGroup

Lessons Learned

  • Numbers on convert are not maintained with precision (13.00 becomes 13.0)
  • When debugging, make a “fat” template if you run into issues. Many “good errors” are hidden by Change Sets and Includes and therefore if you run into weird issues, make a template with everything in it first and then break it out once its working.
  • Make a test blank stack (see above) when trying to develop as a rollback rolls back normally preventing continual stack create/delete cycles
  • Validate all your YAML first and not through a CloudFormation YAML linter, the stricter the YAML linter the better
  • When in doubt, use cfn-flip and see if its still works
  • Develop IAM policies in IAM first, then use a JSON to YAML converter to embed into your templates
  • New features have new bugs, you may call support and want to hit your head on your desk when its something “simple”, but those occurrences are outweighed by the times its a true bug

References

Announcement: https://aws.amazon.com/about-aws/whats-new/2017/03/aws-cloudformation-supports-authoring-templates-with-code-references-and-amazon-vpc-peering/

AWS Documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html