Pushing the CloudFormation Bleeding Edge: Native Modular Templates
When the YAML format for CloudFormation was launched in September 2016, many of the users knew it was only a matter of time until the commonly used pattern of including multiple YAML files into a single file made its way into CloudFormation. On March 28, 2017, AWS did exactly that by launching the AWS::Include Transform, albeit with surprising lack of fanfare.
While YAML was not a prerequisite to having this feature, it made it infinitely easier leverage as an end-user. There are very important things which I have discovered as I integrated AWS::Include into my daily work; some of these are documented fully, others are documented partially, and others not at all.
- Partials - The snippets of CloudFormation stored in S3
- Master - The template executed by the end-user
- Includes - The AWS::Includes Transform
- Your Partials may be in either JSON or YAML
- Partials must use only the long form of a Function Call (e.g Fn::Sub not !Sub)
- Change sets are required for use of Includes
- Partials must be accessible by the end-user’s STS assumption for CloudFormation
- Public ACL Read is not required if you have a good bucket policy
- Partials are included into the Master before evaluation of functions
- Prevents using Fn::Sub in a Location directive for dev/prod s3 path of the Partials
- Errors which occur in Partials create unusual errors on evaluation
- Understanding scope is very important
- Nested Includes calls is not supported (i.e. your partials template cannot have another partial in it)
Now I will dive into some of these key points in detail, use cases for where to AWS::Include, and lessons learned from living on the bleeding edge.
In a future post, I will detail use cases where Includes is ideal for your business such as creating predictable IAM Roles or a multi-engine RDS template
Key Points in Details
The execution model when using Includes is through CloudFormation Change Sets, which is a great way to enforce a known checkpoint but brings in difficulties for people who don’t use CloudFormation daily. When you use a Includes and you want to make a new stack, you are left with two options:
- Create the stack within the AWS Console - the console automatically creates a blank stack, change set, and prompts for CAPABILITY_NAMED_IAM
- Create a “blank” stack (e.g. just a wait handle) and then create a change set against that stack with CAPABILITY_NAMED_IAM in the create-change-set call
Mixing JSON and YAML
This adventure to using Includes was my first significant effort into using YAML for all my CloudFormation, up until this point the need for YAML (specifically inline comments) was not worth the time it would take to rewrite what we had been using to date.
One of the use-cases I leveraged Includes for is deploying IAM Policies and Roles for Federated Login as it requires predictable Role naming. I have found in practice it is easier to write the policies in JSON but as I was doing YAML now, I decided to keep it in pure YAML for readability.
AWS has, thankfully, provided the ability to continue keeping your Partials in either language regardless of the Master’s language. You may choose to keep your JSON templates for things like IAM Policies and use YAML for the Master.
What I did leverage from time to time was cfn-flip to ensure my YAML syntax was inline with the JSON evaluation. As the templates are included they are converted to JSON, so doing this is a reasonable checkpoint for yourself.
Evaluation Logic and Order
As I said above you cannot have any other Function evaluated before the includes happens. This means that you cannot do something like this:
This will throw an error that you must provide a valid S3 URI/Object. I have raised this with support and an RFE has been created to allow this or something like it be accepted.
While the order of operations is not specific to Includes, I was bitten by the unwritten order more than once. For your reference, I have compiled an incomplete list of the evaluation precedence steps here:
- References Lookups
- Conditional Statements
I will try to obtain the information on a complete list of these in the future and make a separate post on that.
The reason I included these is that you are more likely to run into trying to do Mappings inside Conditionals using Reference Lookups and that will fail with unpredictable results.
Errors & Debugging
I have compiled a list of the most common issues I have run into within Includes and YAML in general
|Circular Dependancies||Incorrect/Invalid Reference in the Partial|
|Invalid Policy Syntax for IAM on execution||A
|Invalid MajorEngineVersion ##.# for SqlServer Option Group||SqlServer Option Groups are ##.## and during the YAML to JSON conversion, it drops the superfluous 4th digit, quoting such as
|Must Provide valid S3 URI or S3 Object||1. You are referencing a private S3 Object with no Bucket Policy|
|2. You are trying to do a Function within the S3 Location call|
|3. You are trying to reference a S3 Object which has no valid CloudFormation items|
Scope and Use In Practice
When using Includes, its very important to pay attention to how Scope is in use. You cannot have two Includes calls in a single Scope level. I have detailed interesting use cases around scope below and its important all of these items are mutually exclusive within a single template (#1) or within sections (#2/#3).
1 - Use the transform section of the template
In this example I show how you can replace an entire template with Includes except Parameters, which can not be part of Includes.
2 - A section level
In this example I show how you can take all of the Resources, Outputs, etc of a template and put them into a Partial
3 - Multiple Resources
In this example I show you can use excludes to abstract the body of each resource and output into their own Partial
4 - Within a resource
This example shows how to use Includes to provide some modularity to the end-user while maintaining some attributes which are common in the Partial. When you get to this method, as the warning below notates scope is very tricky and you should take care. In the example I create an RDSDBParameterGroup which allows the Master Template to specify what the Parameters are in use for this RDS. Additionally, I show a second method of advanced scoping which allows you to have an Includes at the end of a section which provides any “generic” items, as a result of scoping, any use of this must be at the end of the Section or it will be overridden. I also demonstrate that regardless of how a resource is declared (e.g. RDSDBParameterGroup is “within a resource”) the Logical ID persists in the template after compilation (e.g the Output for RDSDBParameterGroup is in the Generic Outputs Includes)
Note: This method is considered advanced and requires significant testing
- Numbers on convert are not maintained with precision (13.00 becomes 13.0)
- When debugging, make a “fat” template if you run into issues. Many “good errors” are hidden by Change Sets and Includes and therefore if you run into weird issues, make a template with everything in it first and then break it out once its working.
- Make a test blank stack (see above) when trying to develop as a rollback rolls back normally preventing continual stack create/delete cycles
- Validate all your YAML first and not through a CloudFormation YAML linter, the stricter the YAML linter the better
- When in doubt, use cfn-flip and see if its still works
- Develop IAM policies in IAM first, then use a JSON to YAML converter to embed into your templates
- New features have new bugs, you may call support and want to hit your head on your desk when its something “simple”, but those occurrences are outweighed by the times its a true bug
AWS Documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/create-reusable-transform-function-snippets-and-add-to-your-template-with-aws-include-transform.html