In this article you will learn how deploy your infrastructure as code using AWS Cloudformation. We'll focus on the most important features to develop and customize Cloudformation templates, while learning common patterns for deploying your templates as stacks and integrating them into application automation chains.

This post builds on a previous one where we set up our environment and deployed an AWS Cloudformation stack running a Ghost blog. We'll use sections of this in our examples and examine how they are constructed, so that you can build and run your own template from scratch.

The underlying service that we will be using is AWS Cloudformation, which allows you to declare your entire infrastructure in a JSON or YAML text file. It supports most all AWS services, and is updated to reflect the latest features as they are added. The service supports robust state handling, which makes it the best way to ensure reproducible behavior entirely in code utilizing your CI/CD toolset, the AWS CLI, or the API directly.

In part 2 we will go into some more advanced template techniques as well as how to interact with your stack once it is deployed, using the CLI or a lambda function.


Our Use Case - Deploying the Ghost Blog Platform

The template we used here deploys a set of minimal network resources, an RDS database, and an EC2 instance to run the infrastructure. There is some scripting on the EC2 instance as well which configures the database and launches ghost. Today, we'll walk through the structure of templates, such as the one used in the Ghost blog example. We will also examine the lifecycle of Cloudformation stacks deployed from them, and how stack behavior is influenced by them.

If you do not have an AWS account and the CLI set up, refer to the sections in the previous post, then continue below.


Launching Cloudformation stacks

Command line

To review, once we set up our AWS credentials, we deployed our stack using the CLI:

$ aws cloudformation create-stack \
    --stack-name ghost \
    --region us-east-1 \
    --template-body file://./ghost.template \
    --parameters file://./parameters.json \
    --capabilities CAPABILITY_NAMED_IAM

This kicks off the build of a Cloudformation stack named ghost, in the us-east-1 region, using the cloudformation template located relative to the path from which this is being executed from in the file ghost.template, with input stack parameters configured also externally in a file called parameters.json. We need to include the option to use the capability CAPABILITY_NAMED_IAM because the stack provisions AWS IAM credentials to permit the functionality required by the different AWS resources in the template.

AWS Console

Alternatively, We can create a Cloudformation stack using the AWS console. To do this, navigate to the AWS Cloudformation service within the console and press the Create Stack button. Then, select the template in the Choose File dialog, and click Next.

select_template

After making your template selection, you will be presented with a screen where you can configure all of the options and parameters required create a stack from your template.

configure_parameters_and_options

One benefit of using the console is the UI is aware of AWS types, and inputs are automatically populated with existing resources of those types in your account.

In Your Application's Pipeline

To support more complex Cloudformations stacks, you will want to deploy your Cloudformation templates from CI/CD pipelines.

This can be done natively with AWS Codepipeline using a defined deploy action integration. There are also plugins for Jenkins and Bamboo to accomplish the same task. If you use one of these pipelines, then you'll need to research the options and choose which one best suits your needs.

Directly from an AWS SDK

Finally, you can also create Cloudformation stacks from one of the various AWS language SDKs. Using an application SDK directly would be appropriate for a platform that deploys and hosts infrastructure for others.

Template Structure Explained

Understanding the structure of Cloudformation templates is necessary for customizing your stacks. This section describes the high-level structure and format of a template. If you're already familiar with this, skip to Parameters (insert link)

Cloudformation templates can either be in JSON or YAML format. The main functional difference is that you can add comments with YAML, but can't with JSON. Other than that, the choice between them is a personal preference. I will be using JSON in all of our examples, as I think it is easier to read.

A minimal template might look like this:

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "My Awesome Project",
  "Parameters" : {
    "MyBucketName": {
      "Default": "some-sort-of-bucketname-that-is-unique-enough-to-work",
      "Description": "This is a description of what parameter is",
      "Type": "String"
    }
  },
  "Resources" : {
    "MyBucket": {
        "Type": "AWS::S3::Bucket",
        "Properties": {
          "BucketName": { "Ref": "MyBucketName" }
        }
    }
  }
}

Well-formed JSON

Your template must be a valid JSON document, or it will not work. You can check this using the AWS CLI, which will also give you some feedback on whether it conforms to the valid template spec syntax. If it does, you will get output that looks something like this, returning the parameters:

$ aws cloudformation validate-template --template-body file://./minimal.template --region us-east-1
{
    "Parameters": [
        {
            "ParameterKey": "MyBucketName",
            "DefaultValue": "some-sort-of-bucketname-that-is-unique-enough-to-work",
            "NoEcho": false,
            "Description": "This is a description of what parameter is"
        }
    ],
    "Description": "My Awesome Project"
}

If by mistake we add an extra character which makes the JSON itself invalid (e.g. an extra parenthesis, it gives a JSON formating error:

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: JSON not well-formed. (line 18, column 6)

It helps to run this command often to continually make sure your JSON is valid as you add to your template. It also helps to commit to a source control repo so that you can go back to a known good point if a change you make introduces an error.

AWS Top-level Template Keys/Objects

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "My Awesome Project",
  "Parameters" : {
    ...
  },
  "Resources" : {
    ...
  }
}

In our template, the top level keys have objects as values that hold special meaning. In the template above, we have the keys AWSTemplateFormatVersion, Description, Parameters, and Resources. If there is an extra key that AWS does not recognize, for instance if we added MyData after `Resources above, we get an error:

An error occurred (ValidationError) when calling the ValidateTemplate operation: Invalid template property or properties [MyData]

Resources is the Only Required Top-level Key

Of these top level keys, only Resources is absolutely required. However, pretty much all templates contain these other core top level key sections for the functionality they provide, which we will describe below.

AWSTemplateFormatVersion isn't Required, But is Included by Convention

The string 2010-09-09 is the only valid value for AWSTemplatFormatVersion. Just copy it verbatim in each template. If we change the value to anything but that, we get an error:

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: 2010-09-08 is not a supported value forAWSTemplateFormatVersion.

Include a Description

Description is a free-form string used to describe your template. Be sure to use it and give your stack a meaningful description (better than "My Awesome Project" used above), as this description will show up in describe commands and the console listing for your stack.

Parameters Provide Data to your Template

Use Parameters when you need to pass data inputs to your template that it will use, such as names or instance types.

Resources Contain your Resources

The Resources object is where all of the AWS infrastructure is specified that will be deployed by the template. The Type of each resource needs to be a valid one, otherwise validation will also not pass and you will get an error such as this:

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template format error: Unrecognized resource types: [AWS::S2::Bucket]

All valid types are listed in the AWS Resource and Property Reference, which you should bookmark now.

We will now cover Parameters and Resources in more depth, as they are the most important components of your template.

Parameters - Providing Data to Your Template

Parameters are how you get data into your Cloudformation template. They allow you to define different properties or behavior based on what is passed in without changing anything within the template.

Using Parameters

Parameters can be passed in on the command line or from a separate file. For all but the simplest templates, you will want to use a separate file to provide parameters. If you are using a GUI-based pipeline tool, then your plugin might provide parameter input fields to store them in a database, but typically these plugins take the same format for parameters. Our Ghost Blog template has many parameters contained in a parameters.json.

Parameter Format

When specifying your parameters within your template under the Parameters section, each one must be defined as described in the Parameters documentation:

"Parameters" : {
  "ParameterLogicalID" : {
    "Type" : "DataType",
    "ParameterProperty" : "value"
  }
}

At minimum, you need to give it an alphanumeric ParameterLogicalId (any name, use CamelCaseWithCaps) and a Type. Common types are String or Number. A Number is validated as a number on input, but referenced as a string within your template. These types suffice for most any parameter input needed, but sometimes an AWS-specific type can be useful.

In our example, we used an AWS::EC2::KeyPair::KeyName for a parameter, which will ensure it is a valid KeyPair for the region in which we are deploying the stack. If you're using the UI to deploy your template, AWS-specific types in your parameters will show up as a dropdown with your existing possible resources of that type as input. If you are using a file for parameter input along with the CLI, then using an AWS-specific type will require a string of the resource ID as input.

Parameter Constraints

You can also make sure that parameter inputs meet the criteria you want. These constraints are listed in the ParameterProperty documentation. We use several in our ghost template, which are also described using the ConstraintDescription property.

NoEcho

You will also want to use "NoEcho": "true" for sensitive parameters such as passwords, so that they are replaced with ***** in stack describe operations.

Triggering Conditions

One common usage of parameters is to set a string which can be tested later in the template to conditionally include a property or not. For example, RDS instances can be created by snapshot. If you wanted to create a single template that can create a database from scratch or from using a snapshot, you could define two parameters: UseSnapshot and SnapshotName . In your template, you'd need to define a condition to see if UseSnapshot has a value of true, which, if met, gives your RDS resource a SnapshotName property.

Referencing Parameters

Finally, once you define parameters you can reference them in your template in this way:

"SomeResourcePropertyName": { "Ref": "YourParameterLogicalId" }

"Ref" is just a reference to your parameter; it will be replaced with the string passed in.

Resources - Specify Your Stack

The Resources section is the most important part because this is where you define the AWS infrastructure that you want deployed in a Cloudformation template. Declaring your resources is the secret sauce of 'infrastructure as code'. Most all AWS resources can be deployed with Cloudformation, from the most common, such as AWS::EC2::Instance, to more special purpose service resources, such as the event stream handling facility AWS::Kinesis::Stream.

Format

All resources have the following structure:

"Resources" : {
    "Logical ID" : {
        "Type" : "Resource type",
        "Properties" : {
            ...
        }
    }
}

The Logical ID is a string name you create that describes the resource you want created. It is typically CamelCased with the first letter capitalized. If a resource is the only one of its type in a template, just use the resource type itself as the logical ID. This will make it easier to remember as you refer to the resource elsewhere in your template with References, which are a function that allows us to refer to another resource by Logical ID. The format of a reference is { "Ref": "LogicalIdReferredTo" } References are used often in resources when they require a relation to another resource: for example an S3 Bucket policy needs to refer to an S3 Bucket in order to be useful:

"DropFileBucketPolicy" : {
  "Type" : "AWS::S3::BucketPolicy",
  "Properties" : {
    "Bucket" : {"Ref" : "DropFileBucket"},
    ...
  }
}

Functions can generally be used as values of properties within your template. In the above, Bucket is a property with a value that is a function. The function is a JSON object with the function name as a key --in this case "Ref", and the arguments of the function are the value --here it is just a string, "DropFileBucket". Most functions are of the format "Fn::FunctionName", but "Ref" is the exception and is the most common instrinsic function

The output of the reference function depends on what type of resource you are referring to. The table in the resource and property guide outlines what the output is for each type of resource. It might be the ARN of the deployed resource, or another identifier returned as a string.

The Properties object within a resource contains all of the configuration for a resource. Configurable properties differ per resource type, and are based on the capabilities and options that are configurable for that type. Occasionally, AWS will add new possible configurable properties for a resource as they add features, so use the resource guide link to understand what the current possible properties are for the resource type you are deploying with Cloudformation.

Mappings - In-Template Configuration

While Parameters are used to pass data into your template, the optional Mappings section allows you to provide constants within your template: things that need to be configured and used throughout your template, but don't necessarily need to be passed in and can just exist within the template itself.

Mappings are key/value pairs, but with two levels of keys (key-key-value) to provide a bit more structure to your key/value pairs. Here is an example of how you might use it, from our Ghost Blog template. In it, CidrConfig is our MapName (literal string given to reference the mapping), then GhostVpc is a top level key and Cidr is a second level key.

"Mappings" : {
  "CidrConfig": {
    "GhostVpc": {
      "Cidr": "10.2.0.0/16"
    },
    "PubSubnet": {
      "Cidr": "10.2.3.0/24"
    },
    "DbSubnet1": {
      "Cidr": "10.2.4.0/24"
    },
    "DbSubnet2": {
      "Cidr": "10.2.5.0/24"
    }
  },
}

To refer to a mapping, we use another intrinsic function called FindInMap which returns a string: the value within your mapping that it refers to. This function takes a JSON array as an argument, and expects 3 elements in that array: the MapName, first level key, and second level key

"Resources" : {
  "GhostVpc": {
    "Type": "AWS::EC2::VPC",
    "Properties": {
      "CidrBlock": {
        "Fn::FindInMap": [
          "CidrConfig",
          "GhostVpc",
          "Cidr"
        ]
      },
      "InstanceTenancy": "default"
    }
  }
}

In the above, the intrinsinc function { "Fn::FindInMap": [ "CidrConfig, "GhostVpc", Cidr"] } is replaced with the string in our mapping walking the MapName and two levels of keys to get the value configured. The result after replacement is "CidrBlock": "10.2.0.0/16" within that resource property.

Pseudo Parameter References are often used to reference keys within maps. You can think of pseudo parameters as internal values that every deployed stack has access to, such as the region in which it is being deployed, the account number, etc... These references are very useful to define maps that might need different values for a property based on that information. For example, we might want to use a certain AMI within one region vs. another. In our Ghost Blog template, we have a mapping that does this:

"Mappings: {
  ...
    "AWSRegionArch2AMI" : {
      "ap-northeast-1":{"HVM64":"ami-0f63c02167ca94956"},
      ...
    }
  }
}
      

If the stack is being deployed in the ap-northeast-1 region, we can use { "Fn::FindInMap": [ "AWSREgionArch2AMI", { "Ref": "AWS::Region" }, "HVM64" ] } to get the first mapping value, which is "ami-0f63c02167ca94956".

The AWS::Region and AWS::AccountID pseudo values are particularly useful to use as first or second level keys within mappings to configure different values based on where the Cloudformation template is being deployed.

Summary

Now that we have covered the structure of Cloudformation templates, we can shift our focus to using them. In Part 2, we'll outline the lifecycle of a stack, and how to use it as part of a larger automation chain.