1-888-317-7920 info@2ndwatch.com

AWS re:Invent Keynote Recap – Thursday

Thursday’s General Session Keynote kicked off with Amazon CTO, Werner Vogels, taking the stage to deliver additional product and services announcements with the inclusion of deeper, technical content.  Revisiting his vision for 21st Architectures from the 1st Re:Invent in 2012, Werner focused on what he sees as key guiding principles for next-gen workloads.

  1. Voice represents the next major disruption in computing. Stressing this point, Werner announced the general availability of Alexa for Business to help improve productivity by introducing voice automation into your business.
  2. Use automation to make experimentation easier
  3. Encryption is the ‘key’ to controlling access to your data. As such, encrypting data (at rest and in transit) should be a default behavior.
  4. All the code you should ever write is business logic.


Werner also highlighted the fact that AWS now has over 3,951 new services released since 2012. These services were not built for today but built for the workloads of the future.  The goal for AWS, Werner says, is to be your partner for the future.

One of the highlights of the keynote was when Abby Fuller, evangelist for containers at AWS, came on stage to talk about the future of containers at AWS.  She demoed the use of Fargate which is AWS’s fully managed container service. Think of Fargate as Elastic Beanstalk but for containers.  Per AWS documentation “It’s a technology that allows you to use containers as a fundamental compute primitive without having to manage the underlying instances. All you need to do is build your container image, specify the CPU and memory requirements, define your networking and IAM policies, and launch. With Fargate, you have flexible options to closely match your application needs and you’re billed with per-second granularity.”

The Cloud9 acquisition was also a highlight of the keynote.  Cloud9 is a browser-based IDE for developers.  Cloud9 is completely integrated with AWS and you can create cloud environments, develop code, and push that code to your cloud environment all from within the tool.  It’s really going to be useful for writing and debugging lambda functions for developers that have gone all in on serverless technologies.

New Announcements

AWS Lambda Function Execution Activity Logging – Log all execution activity for your Lambda functions. Previously you could only log events but this allows you to log data events and get additional details.

AWS Lambda Doubles Maximum Memory Capacity for Lambda Functions – You can now allocate 3008MB of memory to your AWS Lambda functions.

AWS Cloud9 –  Cloud9 is a cloud based IDE for writing, running, and debugging your code.

API Gateway now supports endpoint integrations with Private VPCs –  You can now provide access to HTTP(S) resources within your Amazon Virtual Private Cloud (VPC) without exposing them directly to the public Internet.

AWS Serverless Application Repository –   The Serverless Application Repository is a collection of serverless applications published by developers, companies, and partners in the serverless community.

We expect AWS to announce many more awesome features and services before the day ends so stay tuned for our AWS  re:Invent 2017 Products & Services Review and 2017 Conference Recap blog posts for a summary of all of the announcements that are being delivered at AWS re:Invent 2017.


— Brent Clements, Sr. Cloud Consultant, 2nd Watch


A Step-by-Step Guide on Using AWS Lambda-Backed Custom Resources with Amazon CFTs

Amazon CloudFormation Template (CFT) custom resources allow for additional flexibility in building Amazon environments. These can be utilized in a number of ways to enhance automation and make deployment easier. Generally speaking you’ll want to engage custom resources when you need information that is normally not available to the CloudFormation Template in order to complete the processing of the template. This could be a Security Group in another account, or the most updated AMI, or a Spot Price analysis. Additionally it’s useful for creating functionality in CFT that doesn’t exist yet, like verifying that the database size you’ve chosen is valid, or checking if you have a valid password (our example). You won’t want to use it for anything “one-off” as it takes time to develop and process. You will also want to avoid using it for long running processes since AWS CloudFormation will timeout if the internal processes take too long.

To give you an easy example of how this is setup, we’re going to build an AWS Lambda-backed custom resource that will verify that a password is correct by having you type it in twice. If the passwords you type don’t match, the CFT will quit processing and rollback. This is a bit of functionality that’s missing from AWS CFT and can be very frustrating once your environment is deployed and you realize you fat fingered the password parameter. The basic areas we’ll be focusing on are AWS CloudFormation and AWS Lambda. This guide assumes you’re familiar with both of these already, but if you’re not, learn more about AWS Lambda here or AWS CFTs here.

You want to start with the CFT that you’re looking to add the custom resource to and make sure it is functional. It’s always best to start from a place of known good. Adding a Lambda-backed custom resource to a CFT consists of four basic parts:

1. IAM Role for Lambda Execution: This is the role that will be assigned to your Lambda function. You will utilize this role to give the lambda permissions to other parts of AWS as necessary. If you don’t need to add any permissions, just create a role that allows Lambda to write its logs out.

"LambdaExecutionRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Service": ["lambda.amazonaws.com"]
        "Action": ["sts:AssumeRole"]
    "Policies": [{
      "PolicyName": "lambdalogtocloudwatch",
      "PolicyDocument": {
        "Version": "2012-10-17",
        "Statement": [{
          "Effect": "Allow",
          "Action": ["logs:CreateLogGroup",
          "Resource": "arn:aws:logs:*:*:*"

2. The Lambda Function: There are two ways of introducing your Lambda function into your CFT. If it is small, you can embed your function directly into the CFT by using the “ZipFile”
option under the “Code” property of the “AWS::Lambda::Function” resource. Or you can use the “S3Bucket” option and reference an S3 bucket that has your code already present in zip
format. Note that if you use the S3 bucket option the user that deploys the CFT will need permissions to read from the bucket, not the Lambda function. Next you’ll set your “Handler,”
“Runtime,” “Timeout,” and “Role” (which should reference the ARN of the role you created previously). If you are using the ZipFile option, your handler is the default for the runtime.

"CheckPasswordsFunction": {
  "Type": "AWS::Lambda::Function",
  "Properties": {
    "Code": {
      "ZipFile": {
        "Fn::Join": ["\n", [
          "var response = require('cfn-response');",
          "exports.handler = function(event, context) {",
          " if (event.RequestType == 'Delete') {",
          " response.send(event, context, response.SUCCESS);",
          " return;", " }",
          " var password = event.ResourceProperties.Password;",
          " var confpassword = event.ResourceProperties.ConfirmPassword;",
          " var responseData = {};",
          " if (password == confpassword) {",
          " responseData = {'passwordcheck': 'Password Valid!'};",
          " response.send(event, context, response.SUCCESS, responseData);",
          " } else {",
          " responseData = {Error: 'Passwords do not match'};",
          " console.log(responseData.Error);",
          " responseData = {'passwordcheck': 'Password Invalid!'};",
          " response.send(event, context, response.FAILED, responseData);",
          " }", "};"
    "Handler": "index.handler",
    "Runtime": "nodejs",
    "Timeout": "30",
    "Role": {
      "Fn::GetAtt": [
        "LambdaExecutionRole", "Arn"

3. The Lambda Callout: The Lambda callout is where you pass the variables from the CFT to your Lambda function. It’s important to name these appropriately and consider what effect case and naming conventions will have on the runtime you’re using. The “Service Token” property is the ARN of the Lambda function you just created and the rest of the properties are the variables you’re passing through.

"TestPasswords": {
  "Type": "Custom::LambdaCallout",
  "Properties": {
    "ServiceToken": {
      "Fn::GetAtt": [
    "Password": {
      "Ref": "Password"
    "ConfirmPassword": {
      "Ref": "ConfirmPassword"

4. The Response: There are two key parts of the response from the custom resources and this applies to non-Lambda custom resources too. The first is the “Status” of the response. If you return a status of “FAILED,” the CFT will short circuit and rollback. If you return a status of “SUCCESS,” then the CFT will continue to process. This is important because sometimes you’ll want to send SUCCESS even if your lambda didn’t produce the desired result. In the case of our PassCheck, we wanted to stop the CFT from moving forward to save time. Knowing at the end that the password were mismatched would not be very valuable. The second important piece of the response is the “Data.” This is how you pass information back to CloudFormation to process the result. You’ll set the “Data” variable in your code as a json and reference the json key/value pair back inside the CFT. You’ll use the “Fn::GetAtt” option to reference the Lambda callout you created previously and the key of the json data you’re interested in.

"Outputs": {
  "Results": {
    "Description": "Test Passwords Result",
    "Value": {
      "Fn::GetAtt": ["TestPasswords",

As far as your Lambda function is concerned, you may or may not need to reference variables sent from the CloudFormation Template. These variables will be in the “event”->”ResourceProperties” dictionary/hash. For example:


var password = event.ResourceProperties.Password


password = event['ResourceProperties']['Password']

And similarly, once you’re function is completed processing you might need to send a response back to the CFT. Thankfully AWS has created some wrappers to make the response easier. For nodejs it is called “cfn-response” but is only available when using the “ZipFile” option. There is a similar package for Python, but you’ll need to bundle it with your Lambda and deploy from S3. Sending information back from your Lambda is as easy as setting the “Data” variable to a properly formatted json file and sending it back.

if (password == confpassword) {
responseData = {'passwordcheck': 'Password Valid!'};
response.send(event, context, response.SUCCESS, responseData);

That’s it. Creating a Lambda-backed custom resource can add all kinds of additional functionality and options to your CloudFormation Templates. Feel free to download the whole CFT here and it out or use it to learn more, or Contact Us for help in getting started.

-Coin Graham, Sr Cloud Engineer


AWS Lambda Scheduled Event Function Deep Dive: Part 4

Registering the Lambda Function

Now that we’ve created the Lambda function IAM role, for the sake of simplicity, let’s assume the function itself is already written and packaged sitting in an S3 bucket that your Lambda function Role will have access to.  For this example, let’s assume our S3 URL for this backup function is: s3://my-awesome-lambda-functions/ec2_backup_manager.zip

The IAM policy rights required for creating the Lambda function and the Scheduled Event are:

  • lambda:CreateFunction
  • lambda:AddPermission
  • events:PutRule
  • events:PutTargets

Registering the Lambda Function and Scheduled Event via the AWS Lambda API using Python and boto3

Note: Like the previous boto3 example, you must either have your AWS credentials defined in a supported location (e.g. ENV variables, ~/.boto, ~/.aws/configuration, EC2 meta-data) or you must specify credentials when creating your boto3 client (or alternatively ‘session’).  The User/Role associated with the AWS credentials must also have the necessary rights, defined by policy, to perform the required operations against the AWS Lambda API.

A few notes on the creation_function function arguments
  • The Runtime is the language and version being used by our function (i.e. python2.7)
  • The Role is the ARN of the role we created in the previous exercise
  • The Handler is the function within your code that Lambda calls to begin execution. For our python example the value is: {lambda_function_name}.{handler_function_name}
  • The Code is either a base64 encoded zip file or the s3 bucket and key
An important Warning about IAM role replication delay

In the previous step we created an IAM role that we reference in the code below when creating our Lambda function.  Since IAM is an AWS region independent service it takes some time – usually less than a minute – for create/update/delete actions against the IAM API to replicate to all AWS regions.  This means that when we perform the create_function operation in the script below, we may very well receive an error from the Lambda API stating that the role we specified does not exist.  This is especially true when scripting an operation where the create_function operation happens only milliseconds after the create_role function.  Since there is really no way of querying the Lambda API to see if the role is available in the region yet prior to creating the function, the best option is to use exception handling to catch the specific error where the role does not yet exist and wrap that exception handler in a retry loop with an exponential back-off algorithm (though sleeping for 10-15 seconds will work just fine too).

Let’s pick up in the python script where we previously left off in the last example:


Registering the Lambda Function and Scheduled Event using AWS CloudFormation Template

Note: The S3Bucket MUST exist in the same region you are launching your CFT stack in.  To support multi-region templates I generally will have a Lambda S3 bucket for each Lambda region and append a .{REGION_NAME} suffix to the bucket name (e.g. my-awesome-lambda-functions.us-west-2).  Since CloudFormation provides us with a psuedo-parameter of the region you are launching the stack in (AWS::Region), you can utilize that to ensure you are referencing the appropriate bucket (see my example below).

The following block of JSON can be used in conjunction with our previous CloudFormation snippet by being added to the template’s “Resource” section to create the Lambda function, CloudWatch Event, and Input:


If you implement your Lambda functions using either of the two examples provided you should be able to reliably create, , manage, automate and scale them to whatever extent and whatever schedule you need.  And the best part is you will only be charged for the ACTUAL compute time you use, in 100ms increments.

Now go have fun, automate everything you possibly can, and save your organization thousands of dollars in wasted compute costs!  Oh, and conserve a bunch of energy while you’re at it.

-Ryan Kennedy, Sr Cloud Consultant


AWS Lambda Scheduled Event Function Deep Dive: Part 3

Creating the Lambda Function IAM Role

In our last article, we looked at how to set up scheduled events using either the API (python and boto3) or CloudFormation, including the required Trusted Entity Policy Document and IAM Role. This role and policy can be created manually using the AWS web console (not recommended), scripted using the IAM API (e.g. Python and boto3), or using a templating tool (e.g. CloudFormation. Hashicorp’s Terraform).  For this exercise we will cover both the scripted and the template tool approaches.

The IAM policy rights required for creating the Lambda function role and policy are:

  • iam:CreateRole
  • iam:PutRolePolicy

Creating the Lambda IAM role via the IAM API using Python and boto3

Note: For the following example you must either have your AWS credentials defined in a supported location (e.g. ENV variables, ~/.boto, ~/.aws/configuration, EC2 meta-data) or you must specify credentials when creating your boto3 client (or alternatively ‘session’).  The User/Role associated with the AWS credentials must also have the necessary rights, defined by policy, to perform the required operations against AWS IAM API.

The following python script will produce our desired IAM role and policy:


That will create the necessary lambda function role and its inline policy.

Creating the Lambda IAM role using AWS CloudFormation Template

The following block of JSON can be added to a CloudFormation template’s “Resource” section to create the Lambda function role and its inline policy:


Visit us next week for the final segment of this blog series – Registering the Lambda Function.

-Ryan Kennedy, Sr Cloud Consultant


AWS Lambda Scheduled Event Function Deep Dive: Part 2

Down to the “Nitty-Gritty”

Now that we have an understanding of how AWS Lambda scheduled events can be expressed, we can dive into a real-world scenario and examine how to set that up using either the API (python and boto3) or CloudFormation.  Because what fun would it be doing it in the web console after all?  And this is a deep-dive, so using the web console would be a distasteful choice anyway.  Also…automation.  Enough said.

Suffice to say, creating the scheduled event for your function in the AWS web console can be done quite easily by selecting “Scheduled Event” from the “Event source type” drop-down list and defining your expression.

The Use Case

Let’s assume we’ve written a nice little Lambda function that will search the EC2 API and find all of our instances running in a region, or multiple regions, and manage EBS snapshot backups for all EBS volumes on any instances with a specific tag.

We could hard-code our parameters in the Lambda function or derive them a number of ways, but let’s assume we’ve done the right thing and have specified them in the Scheduled Event Input parameter.  You might say, “Yeah, but I can just derive the current AWS region during the Lambda function execution, so why bother even providing that input?”  To which I would say:

Lambda is only currently available in the us-east-1, us-west-2, ap-northeast-1, eu-west-1, and eu-central-1 regions. What if you want to manage EC2 backups in ap-southeast-2 or another region where lambda isn’t yet available?

Defining our Inputs

backup_tag (dictionary): A single key/value pair used by the backup function to identify which EC2 instance (any instances with a matching tag) to manage backups on.
Ex. { “Key”: “Environment”, “Value”: “Prod” }

regions (list of strings): A list of region(s) to manage EC2 backups against.
Ex. [ “us-west-2”, “us-west-1”, “us-east-1” ]

support_email (string): An email address to send backup reports, alerts, etc. to.
Ex. “backup.admin@2ndwatch.com”

Those three inputs would be captured as a JSON string like so:


Lambda Function IAM Role Requirement

Lambda functions require an IAM Role be specified at time of creation.  The Role must have the lambda service added as a “Trusted Entity” so that it can assume the Role.  The Trusted Relationship Policy Document (called AssumeRolePolicyDocument in CloudFormation) should look like this:


In addition to the Trusted Entity Policy Document, the Role should have a policy, inline or managed, assigned to it that will allow the Lambda function all of the access it needs to AWS resources and APIs (e.g. EC2 describe instances, create snapshots, delete snapshots).  For our use-case the following policy is a good place to start:


Come back later this week for part 3 of this blog series – Creating the Lambda Function IAM Role.

-Ryan Kennedy, Sr Cloud Consultant