1-888-317-7920 info@2ndwatch.com

Standardizing & Automating Infrastructure Development Processes

Introduction

Let’s start with a small look at the current landscape of technology and how we arrived here. There aren’t very many areas of tech that have not been, or are not currently, in a state of fluctuation. Everything from software delivery vehicles and development practices, to infrastructure creation has experienced some degree of transformation over the past several years. From VMs to Containers, it seems like almost every day the technology tool belt grows a little bigger, and our world gets a little better (though perhaps more complex) due to these advancements. For me, this was incredibly apparent when I began to delve into configuration management which later evolved into what we now call “infrastructure as code”.

The transformation of the development process began with simple systems that we once used to manage a few machines (like bash scripts or Makefiles) which then morphed into more complex systems (CF Engine, Puppet, and Chef) to manage thousands of systems. As configuration management software became more mature, engineers and developers began leaning on them to do more things. With the advent of hypervisors and the rise of virtual machines, it was only a short time before hardware requests changed to API requests and thus the birth of infrastructure as a service (IaaS). With all the new capabilities and options in this brave new world, we once again started to lean on our configuration management systems—this time for provisioning, and not just convergence.

Provisioning & Convergence

I mentioned two terms that I want to clarify; provisioning and convergence. Say you were a car manufacturer and you wanted to make a car. Provisioning would be the step in which you request the raw materials to make the parts for your automobile. This is where we would use tools like Terraform, CloudFormation, or Heat. Whereas convergence is the assembly line by which we check each part and assemble the final product (utilizing config management software).

By and large, the former tends to be declarative with little in the way of conditionals or logic, while the latter is designed to be robust and malleable software that supports all the systems we run and plan on running. This is the frame for the remainder of what we are going to talk about.

By separating the concerns of our systems, we can create a clear delineation of the purpose for each tool so we don’t feel like we are trying to jam everything into an interface that doesn’t have the most support for our platform or more importantly our users. The remainder of this post will be directed towards the provisioning aspect of configuration management.

Standards and Standardization

These are two different things in my mind. Standardization is extremely prescriptive and can often seem particularly oppressive to professional knowledge workers, such as engineers or developers. It can be seen as taking the innovation away from the job. Whereas standards provide boundaries, frame the problem, and allow for innovative ways of approaching solutions. I am not saying standardization in some areas is entirely bad, but we should let the people who do the work have the opportunity to grow and innovate in their own way with guidance. The topic of standards and standardization is part of a larger conversation about culture and change. We intend to follow up with a series of blog articles relating to organizational change in the era of the public cloud in the coming weeks.

So, let’s say that we make a standard for our new EC2 instances running Ubuntu. We’ll say that all instances must be running the la official Canonical Ubuntu 14.04 AMI and must have these three tags; Owner, Environment, and Application. How can we enforce that in development of our infrastructure? On AWS, we can create AWS Config Rules, but that is reactive and requires ad-hoc remediation. What we really want is a more prescriptive approach bringing our standards closer to the development pipeline. One of the ways I like to solve this issue is by creating an abstraction. Say we have a terraform template that looks like this:

# Create a new instance of the la Ubuntu 14.04 on an
provider "aws" { region = "us-west-2"
}

data "aws_ami" "ubuntu" { most_recent = true

filter {
name	= "name" values =
["ubuntu/images/hvm-ssd/ubuntu-trusty-1 4.04-amd64-server-*"]
}

filter {
name	= "virtualization-type" values = ["hvm"]
}

owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" { ami	=
"${data.aws_ami.ubuntu.id}" instance_type = "t2.micro"

tags {
Owner	= "DevOps Ninja" Environment = "Dev" Application = "Web01"
}
}

This would meet the standard that we have set forth, but we are relying on the developer or engineer to adhere to that standard. What if we enforce this standard by codifying it in an abstraction? Let’s take that existing template and turn it into a terraform module instead.

Module

# Create a new instance of the la Ubuntu 14.04 on an

variable "aws_region" {} variable "ec2_owner" {} variable "ec2_env" {} variable "ec2_app" {}
variable "ec2_instance_type" {}

provider "aws" {
region = "${var.aws_region}"
}

data "aws_ami" "ubuntu" { most_recent = true

filter {
name	= "name" values =
["ubuntu/images/hvm-ssd/ubuntu-trusty-1 4.04-amd64-server-*"]
}

filter {
name	= "virtualization-type" values = ["hvm"]
}

owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" { ami	=
"${data.aws_ami.ubuntu.id}" instance_type =
"${var.ec2_instance_type}"

tags {
Owner	= "${var.ec2_owner}" Environment = "${var.ec2_env}" Application = "${var.ec2_app}"
}
}

Now we can have our developers and engineers leverage our tf_ubuntu_ec2_instance module.

New Terraform Plan

module "Web01" { source =
"git::ssh://git@github.com/SomeOrg/tf_u buntu_ec2_instance"

aws_region = "us-west-2" ec2_owner = "DevOps Ninja" ec2_env	= "Dev"
ec2_app	= "Web01"
}

This doesn’t enforce the usage of the module, but it does create an abstraction that provides an easy way to maintain standards without a ton of overhead, it also provides an example for further creation of modules that enforce these particular standards.

This leads us into another method of implementing standards but becomes more prescriptive and falls into the category of standardization (eek!). One of the most underutilized services in the AWS product stable has to be Service Catalog.

AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures. AWS Service Catalog allows you to centrally manage commonly deployed IT services, and helps you achieve consistent governance and meet your compliance requirements, while enabling users to quickly deploy only the approved IT services they need.

The Interface

Once we have a few of these projects in place (e.g. a service catalog or a repo full of composable modules for infrastructure that meet our standards) how do we serve them out? How you spur adoption of these tools and how they are consumed can be very different depending on your organization structure. We don’t want to upset workflow and how work gets done, we just want it to go faster and be more reliable. This is what we talk about when we mention the interface. Whichever way work flows in, we should supplement it with some type of software or automation to link those pieces of work together. Here are a few examples of how this might look (depending on your organization):

1.) Central IT Managed Provisioning

If you have an organization that manages requests for infrastructure, having this new shift in paradigm might seem daunting. The interface in this case is the ticketing system. This is where we would create an integration with our ticketing software to automatically pull the correct project from service catalog or module repo based on some criteria in the ticket. The interface doesn’t change but is instead supplemented by some automation to answer these requests, saving time and providing faster delivery of service.

2.) Full Stack Engineers

If you have engineers that develop software and the infrastructure that runs their applications this is the easiest scenario to address in some regards and the hardest in others. Your interface might be a build server, or it could simply be the adoption of an internal open source model where each team develops modules and shares them in a common place, constantly trying to save time and not re-invent the wheel.

Supplementing with software or automation can be done in a ton of ways. Check out an example Kelsey Hightower wrote using Jira.

“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” – John Gall

All good automation starts out with a manual and well-defined process. Standardizing & automating infrastructure development processes begins with understanding how our internal teams can be best organized.  This allows us to efficiently perform work before we can begin automating. Work with your teammates to create a value stream map to understand the process entirely before doing anything towards the efforts of automating a workflow.

With 2nd Watch designs and automation you can deploy quicker, learn faster and modify as needed with Continuous Integration / Continuous Deployment (CI/CD). Our Workload Solutions transform on-premises workloads to digital solutions in the public cloud with next generation products and services.  To accelerate your infrastructure development so that you can deploy faster, learn more often and adapt to customer requirements more effectively, speak with a 2nd Watch cloud deployment expert today.

– Lars Cromley, Director of Engineering, Automation, 2nd Watch

Facebooktwittergoogle_pluslinkedinmailrss

AWS CFT vs. Terraform: Advantages and Disadvantages

UPDATE:  AWS Cloudformation now supports YAML.  To be sure, this is a huge improvement over JSON in terms of formatting and use of comments.  This will also simplify windows and linux userdata scripts.  So for teams that are just starting with AWS and don’t need any of the additional benefits of Terraform, YAML would be the best place to start.  Existing teams will likely still have a cache of JSON templates that they will need to recreate and should consider whether the other benefits of Terraform warrant a move away from CFT.

If you’re familiar with AWS CloudFormation Templates (CFTs) and how they work but have been considering Terraform, this guide is for you.  This basic guide will introduce you to some of the advantages and disadvantages of Terraform in comparison to CFT to determine if you should investigate further and try it yourself.  If you don’t have a rudimentary familiarity with Terraform, head over to https://www.terraform.io/intro/index.html for a quick overview.

Advantages

Formatting – This is far and away the strongest advantage of Terraform.  JSON is not a coding language, and it shows.  It’s common for CFTs to be 3000 lines long, and most of that is just JSON braces and bracket.  Terraform has a simple (but custom) HCL for creating templates and makes it easy to document and comment your code.  Whole sections can be moved to a folder structure for design and clarity.  This makes your infrastructure feel a bit more like actual code.  Lastly, you won’t need to convert Userdata bash and PowerShell scripts to JSON only to deploy and discover you forgot one last escaping backslash.  Userdata scripts can be written in separate files exactly as you would write them on the server locally.  For an example, here’s a comparison of JSON to Terraform for creating an instance:

Instance in CFT


"StagingInstance": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    "UserData": {
      "Fn::Base64": {
        "Fn::Join": ["", [
          "#!/bin/bash -v\n",
          "yum update -y aws*\n",
          "yum update --sec-severity=critical -y\n",
          "yum install -y aws-cfn-bootstrap\n",
          "# download data and install file\n",
          "/opt/aws/bin/cfn-init -s ", {
            "Ref": "AWS::StackName"
          }, " -r StagingInstance ",
          "    --region ", {
            "Ref": "AWS::Region"
          },
          " || error_exit 'Failed to run cfn-init'\n"
        ]]
      }
    },
    "SecurityGroupIds": [{
      "Ref": "StagingSecurityGroup"
    }],
    "ImageId": {
      "Ref": "StagingAMI"
    },
    "KeyName": {
      "Ref": "InstancePrivateKeyName"
    },
    "InstanceType": {
      "Ref": "StagingInstanceType"
    },
    "IamInstanceProfile": {
      "Ref": "StagingInstanceProfile"
    },
    "Tags": [{
      "Key": "Name",
      "Value": {
        "Fn::Join": ["-", [
          "staging", {
            "Ref": "AWS::StackName"
          }, "app-instance"
        ]]
      }
    }],
    "SubnetId": {
      "Ref": "PrivateSubnet1"
    }
  }
}

Instance in Terraform


#
Create the staging instance
resource "aws_instance"
"staging" {
  ami = "${var.staging_instance_ami}"
  instance_type =
    "${var.staging_instance_type}"
  subnet_id =
    "${var.private_subnet_id_1}"
  vpc_security_group_ids = [
    "${aws_security_group.staging.id}"
  ]
  iam_instance_profile =
    "${aws_iam_instance_profile.staging.name}"
  key_name =
    "${var.instance_private_key_name}"
  tags {
    Name =
      "staging-${var.stack_name}-instance"
  }
  user_data = "${file("
  instances / staginguserdatascript.sh ")}"
}

Managing State – This is the second advantage for Terraform.  Terraform knows the state of the environment from the last run, so you can run “terraform plan” and see exactly what has changed with the items that Terraform has created.  With an update to a CFT, you only know that an item will be “Modified,” but not how.  At that point you’ll need to audit the modified item and manually compare to the existing CFT to determine what needs to be updated.

Multi-Provider Support – Depending on how you utilize AWS and other providers, this can be a very big deal.  Terraform gives you a centralized location to manage multiple providers.  Maybe your DNS is in Azure but your servers are in AWS.  You could build an ELB and update the Azure DNS all in the same run.  Or maybe you want to update your AWS infrastructure and also update your DataDog monitoring too.  If you needed a provider they didn’t have, you could presumably add it since the code is open source.

Short learning curve – While they did introduce custom formatting for Terraform templates, the CFT and API nomenclature is *mostly* preserved.  For example, when creating an instance in CFT you need an InstanceType and KeyName. In Terraform this is instance_type and key_name.  Words are separated by underscores and all lowercase.  This makes it somewhat easy to migrate existing CFTs.  All told, it took about a day of experimentation with Terraform to feel comfortable.

Open Source – The general terraform tool is open source, which brings all the good and bad to the table that you normally associate with open source.  As mentioned previously, if you have GoLang resources, the world is your oyster.  Terraform can be made to do whatever you want it to do, and adding back to the repository will enhance it for everyone else.  You can check out the git repo to see that it has pretty active development.

Challenges

Cost – The free version of Terraform is free, but the enterprise version is expensive.  Of course the enterprise version adds a lot of bells and whistles, but I would recommend doing a serious evaluation to determine if they are worth the cost.

No Rollback – Rolling back a CFT deployment or upgrade is sometimes a blessing and sometimes a curse, but with CFT at least you have an option.  With Terraform, there is never an automatic rollback.  You have to figure out what went wrong and plow forward, or first rollback your code then re-deploy.  Either way it can be messy.  However, rollback for AWS CFT can be messy too.  Especially when changes are introduced that make CFT deployment and reconfiguration incompatible.  This invariably leads to the creation of an AWS support ticket to make adjustments to the CFT that is not possible otherwise.

CFT is “tightly coupled” with AWS, while Terraform is not.  This is the YANG to the open source YIN.  Amazon has a dedicated team to continue to improve and update CFTs.  They won’t just focus on the most popular items and will have access to internal resources to vet and prove out their approach.

Conclusion

While this article only scratches the surface of the differences between utilizing AWS CFT and Terraform, it provides a good starting point when evaluating both.  If you’re looking for a better “infrastructure as code,” state management, or multi-provider support, Terraform is definitely worth a look.  We are here to help our customers, so if you need help developing a cloud-first strategy, contact us here.

-Coin Graham, Sr Cloud Consultant, 2nd Watch

Facebooktwittergoogle_pluslinkedinmailrss