1-888-317-7920 info@2ndwatch.com

Standardizing & Automating Infrastructure Development Processes

Introduction

Let’s start with a small look at the current landscape of technology and how we arrived here. There aren’t very many areas of tech that have not been, or are not currently, in a state of fluctuation. Everything from software delivery vehicles and development practices, to infrastructure creation has experienced some degree of transformation over the past several years. From VMs to Containers, it seems like almost every day the technology tool belt grows a little bigger, and our world gets a little better (though perhaps more complex) due to these advancements. For me, this was incredibly apparent when I began to delve into configuration management which later evolved into what we now call “infrastructure as code”.

The transformation of the development process began with simple systems that we once used to manage a few machines (like bash scripts or Makefiles) which then morphed into more complex systems (CF Engine, Puppet, and Chef) to manage thousands of systems. As configuration management software became more mature, engineers and developers began leaning on them to do more things. With the advent of hypervisors and the rise of virtual machines, it was only a short time before hardware requests changed to API requests and thus the birth of infrastructure as a service (IaaS). With all the new capabilities and options in this brave new world, we once again started to lean on our configuration management systems—this time for provisioning, and not just convergence.

Provisioning & Convergence

I mentioned two terms that I want to clarify; provisioning and convergence. Say you were a car manufacturer and you wanted to make a car. Provisioning would be the step in which you request the raw materials to make the parts for your automobile. This is where we would use tools like Terraform, CloudFormation, or Heat. Whereas convergence is the assembly line by which we check each part and assemble the final product (utilizing config management software).

By and large, the former tends to be declarative with little in the way of conditionals or logic, while the latter is designed to be robust and malleable software that supports all the systems we run and plan on running. This is the frame for the remainder of what we are going to talk about.

By separating the concerns of our systems, we can create a clear delineation of the purpose for each tool so we don’t feel like we are trying to jam everything into an interface that doesn’t have the most support for our platform or more importantly our users. The remainder of this post will be directed towards the provisioning aspect of configuration management.

Standards and Standardization

These are two different things in my mind. Standardization is extremely prescriptive and can often seem particularly oppressive to professional knowledge workers, such as engineers or developers. It can be seen as taking the innovation away from the job. Whereas standards provide boundaries, frame the problem, and allow for innovative ways of approaching solutions. I am not saying standardization in some areas is entirely bad, but we should let the people who do the work have the opportunity to grow and innovate in their own way with guidance. The topic of standards and standardization is part of a larger conversation about culture and change. We intend to follow up with a series of blog articles relating to organizational change in the era of the public cloud in the coming weeks.

So, let’s say that we make a standard for our new EC2 instances running Ubuntu. We’ll say that all instances must be running the la official Canonical Ubuntu 14.04 AMI and must have these three tags; Owner, Environment, and Application. How can we enforce that in development of our infrastructure? On AWS, we can create AWS Config Rules, but that is reactive and requires ad-hoc remediation. What we really want is a more prescriptive approach bringing our standards closer to the development pipeline. One of the ways I like to solve this issue is by creating an abstraction. Say we have a terraform template that looks like this:

# Create a new instance of the la Ubuntu 14.04 on an
provider "aws" { region = "us-west-2"
}

data "aws_ami" "ubuntu" { most_recent = true

filter {
name	= "name" values =
["ubuntu/images/hvm-ssd/ubuntu-trusty-1 4.04-amd64-server-*"]
}

filter {
name	= "virtualization-type" values = ["hvm"]
}

owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" { ami	=
"${data.aws_ami.ubuntu.id}" instance_type = "t2.micro"

tags {
Owner	= "DevOps Ninja" Environment = "Dev" Application = "Web01"
}
}

This would meet the standard that we have set forth, but we are relying on the developer or engineer to adhere to that standard. What if we enforce this standard by codifying it in an abstraction? Let’s take that existing template and turn it into a terraform module instead.

Module

# Create a new instance of the la Ubuntu 14.04 on an

variable "aws_region" {} variable "ec2_owner" {} variable "ec2_env" {} variable "ec2_app" {}
variable "ec2_instance_type" {}

provider "aws" {
region = "${var.aws_region}"
}

data "aws_ami" "ubuntu" { most_recent = true

filter {
name	= "name" values =
["ubuntu/images/hvm-ssd/ubuntu-trusty-1 4.04-amd64-server-*"]
}

filter {
name	= "virtualization-type" values = ["hvm"]
}

owners = ["099720109477"] # Canonical
}

resource "aws_instance" "web" { ami	=
"${data.aws_ami.ubuntu.id}" instance_type =
"${var.ec2_instance_type}"

tags {
Owner	= "${var.ec2_owner}" Environment = "${var.ec2_env}" Application = "${var.ec2_app}"
}
}

Now we can have our developers and engineers leverage our tf_ubuntu_ec2_instance module.

New Terraform Plan

module "Web01" { source =
"git::ssh://git@github.com/SomeOrg/tf_u buntu_ec2_instance"

aws_region = "us-west-2" ec2_owner = "DevOps Ninja" ec2_env	= "Dev"
ec2_app	= "Web01"
}

This doesn’t enforce the usage of the module, but it does create an abstraction that provides an easy way to maintain standards without a ton of overhead, it also provides an example for further creation of modules that enforce these particular standards.

This leads us into another method of implementing standards but becomes more prescriptive and falls into the category of standardization (eek!). One of the most underutilized services in the AWS product stable has to be Service Catalog.

AWS Service Catalog allows organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multi-tier application architectures. AWS Service Catalog allows you to centrally manage commonly deployed IT services, and helps you achieve consistent governance and meet your compliance requirements, while enabling users to quickly deploy only the approved IT services they need.

The Interface

Once we have a few of these projects in place (e.g. a service catalog or a repo full of composable modules for infrastructure that meet our standards) how do we serve them out? How you spur adoption of these tools and how they are consumed can be very different depending on your organization structure. We don’t want to upset workflow and how work gets done, we just want it to go faster and be more reliable. This is what we talk about when we mention the interface. Whichever way work flows in, we should supplement it with some type of software or automation to link those pieces of work together. Here are a few examples of how this might look (depending on your organization):

1.) Central IT Managed Provisioning

If you have an organization that manages requests for infrastructure, having this new shift in paradigm might seem daunting. The interface in this case is the ticketing system. This is where we would create an integration with our ticketing software to automatically pull the correct project from service catalog or module repo based on some criteria in the ticket. The interface doesn’t change but is instead supplemented by some automation to answer these requests, saving time and providing faster delivery of service.

2.) Full Stack Engineers

If you have engineers that develop software and the infrastructure that runs their applications this is the easiest scenario to address in some regards and the hardest in others. Your interface might be a build server, or it could simply be the adoption of an internal open source model where each team develops modules and shares them in a common place, constantly trying to save time and not re-invent the wheel.

Supplementing with software or automation can be done in a ton of ways. Check out an example Kelsey Hightower wrote using Jira.

“A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.” – John Gall

All good automation starts out with a manual and well-defined process. Standardizing & automating infrastructure development processes begins with understanding how our internal teams can be best organized.  This allows us to efficiently perform work before we can begin automating. Work with your teammates to create a value stream map to understand the process entirely before doing anything towards the efforts of automating a workflow.

With 2nd Watch designs and automation you can deploy quicker, learn faster and modify as needed with Continuous Integration / Continuous Deployment (CI/CD). Our Workload Solutions transform on-premises workloads to digital solutions in the public cloud with next generation products and services.  To accelerate your infrastructure development so that you can deploy faster, learn more often and adapt to customer requirements more effectively, speak with a 2nd Watch cloud deployment expert today.

– Lars Cromley, Director of Engineering, Automation, 2nd Watch

Facebooktwittergoogle_pluslinkedinmailrss

VPC Peering via CloudFormation

Here is the use case:   Imagine you have several VPCs (target VPCs) that you want to attach to a services VPC.  In the services VPC you have an application license server, or a Domain Controller, or any other workload that instances in each of the other target VPCs need access to.  Every time a new target VPC is created you want to have an easy way to connect that VPC to the services VPC.  You also want to use CloudFormation for creating the peer between the target VPCs and the services VPC because it’s easier for you to manage and keep track of the peering connections if they are tied to a CloudFormation stack.

In the picture below, the red arrows represent the target VPCs and the green arrow represents the services VPC.  Here we’ll discuss a template that will connect VPC A (services VPC) to VPC G (a target VPC).  This template, with slight modifications, could be used for attaching any target VPC (B – G) with the services VPC.

Image1

The requirement for a template that would do this would be one that created the following resources;

  • A VPC Peering Connection
  • A VPC route from the services VPC to the target VPC
  • A VPC route from the target VPC to the services VPC.

The two VPC routes listed above could be several routes depending on your routing requirements.  Say you had several route tables – one for public subnets and one for private subnets – you would want to account for all of them in the template to make sure they know the proper route to the services VPC.

In this example, since both VPCs (the target and the service VPC) have already been created, the parameters in your CloudFormation template should account for inputting these values.  Here we have parameters for the services VPC as well as the target VPC.

Image2

Expanding the parameters section we see that the parameter we specified as “ServicesVPC” and “targetVPCtoPeerto” are actually VPC IDs that will be used in the resource section of our template.

Image3

Setting these two values as input parameters will let you use this template to theoretically connect any two VPCs together as long as you can provide the VPC IDs.  If we were to expand the “RouteTableForTargetVPCYouArePeeringto” we would see the parameter is actually a route table ID.

Image4

Since this is a simplified example, we are only peering two VPCs that have a single route table.  If we wanted to peer VPCs that had additional route tables, say you had one for private traffic and one for public traffic, you would just add another parameter for the additional route table and tie that to a corresponding resource in the resource section of the template.  It’s also important to remember that the parameters can be created and declared but don’t necessarily have to be used or referenced in your template.  Additionally, you can see “RouteTableForServicesVPC” is pre-filled because we already are using it with existing peers.

Moving down to the resources section of the template we see that the only resources that need to be created would be the VPC Peering Connection, a VPC route from the services VPC to the target VPC, and a VPC route from the target VPC to the services VPC.

Image5

Expanding the resources, we can see that the VPC peering connection will get created and will reference our two VPC ID parameters.  Additionally, we will have routes created and assigned to each route table for each VPC.   The route has the hard coded “Destination CIDR Block” and references the “VPC Peering Connection ID” created above it.  You could very easily strip out the hard coded “Destination CIDR block” and use it as an input parameter as well to give you even more flexibility.

Finally, once this template is run, as long as you put in each of the correct parameters, you will have two VPCs that are connected together via VPC peering.

Image6

Remember to adjust the security groups to allow traffic to pass between the VPCs.  Also remember the rules of VPC peering.  You can see those at  http://docs.aws.amazon.com/AmazonVPC/la/PeeringGuide/vpc-peering-overview.html

Please leave a note if you have any comments/question, or you can contact 2nd Watch for all your CloudFormation needs!

-Derek Baltazar, Cloud Architect

Facebooktwittergoogle_pluslinkedinmailrss

AWS Launches Visual App for CloudFormation

New visualization tool is the first native visualization for AWS infrastructure, and a groundbreaking development in the adoption of infrastructure as a code.

Cloud developers and architects use AWS CloudFormation to design, launch and update an AWS application or service architecture stack for repeatable workloads. CloudFormation provides templates for creating an entire environment, such as a website, using the JSON scripting language. Developers don’t need to figure out the order for provisioning AWS services or worry about the dependencies, as the free tool handles all the behind the scene configurations.

This is a huge help when you have workloads that must launch over and over again, saving time on provisioning and configuration. You can actually launch “with the click of a button.” On the other hand, CloudFormation files typically consist of thousands of lines of code, which doesn’t make them easy to modify or share.

Now, with the new AWS CloudFormation Designer, customers can view the details behind an AWS environment through a simpler, graphical view. You don’t need JSON expertise to collaborate on design and planning. The visual drag-and-drop interface is remarkably easy to use, compared with how cloud developers have been working thus far. Instead of writing several lines of code, you can draw a line on the screen connecting one resource to another.

The big picture

AWS CloudFormation Designer will expand the universe of IT people who can write AWS scripts and manage workloads, since they won’t need to know JSON. That means more hands on deck for new projects. After you create the visualization in the tool, you can launch the environment right then and there. We’ve seen that CloudFormation Designer reduces the time to create and launch a new workload from hours to minutes.

What’s also intriguing about this new feature is that it further affirms the infrastructure-as-code mindset. Consider the impact that tools like Visual Studio, Borland and IDEs have had on Microsoft .NET and Java developers. Developers can write code faster and reduce errors. Similarly, AWS has in effect created an interactive development environment (IDE) type tool for the cloud. This is a radical departure for the public cloud leader; AWS is known for its boxes and pipes, not sophisticated visual tools.

AWS CloudFormation Designer is a refreshing development for those who believe public cloud infrastructure will take over the world. One of the challenges with cloud computing adoption is that infrastructure as a service remains a new language to many corporate IT departments. AWS has made a smart move here by introducing more transparency to its platform, and opening the door for IT generalists to get involved. We’re excited that our customers and other companies will benefit from this new view of AWS.

-Kris Bliesner, Founder and CTO

Facebooktwittergoogle_pluslinkedinmailrss

Cloud Analytics – What's in Your Cloud?

One of the things “everyone knows” about migrating to the Cloud is that it saves companies money. Now you don’t need all those expensive data centers with the very physical costs associated with them. So companies migrate to the Cloud and are so sure they will see their costs plummet… then they get their bill for Cloud usage and experience sticker shock. Typically, this is when our customers reengage 2nd Watch – they ask us why it costs so much, what they can do to decrease their costs, and of course everyone’s favorite – why didn’t you tell me it would be so much?

First, in order to know why you are spending so much you need to analyze your environment. I’m not going to go into how Amazon bills and walk you through your entire bill in this blog post. That’s something for another day perhaps. What I do want to look into is how to enable you to see what you have in your Cloud.

Step one: tag it! Amazon gives you the ability to tag almost everything in your environment, including ELB’s, which was most recently added. I always highly recommend to my customers to make use of this feature. Personally, whenever I create something manually or programmatically I add tags to identify what it is, why it’s there, and of course who is paying for it. Even in my sandbox environment, it’s a way to tell colleagues “Don’t delete my stuff!” Programmatically, tags can be added through CloudFormation, Elastic Beanstalk, auto scaling, CLI, as well as third party tools like Puppet and Chef. From a feature perspective, there are very few AWS components that don’t support tags, and more are constantly being added.

That’s all well and good, but how does this help analytics? Tagging is actually is the basis for pretty much all analytics, and without it you have to work much harder for far less information. For example, I can tag EC2 instances to indicate applications, projects, or environments. I can then run reports that look for specific tags – how many EC2 instances are associated with Project X and what are the instance types? What are the business applications using my various RDS instances? – and suddenly when you get your bill, you have the ability to determine who is spending money in your organization and work with them on spending it smartly.

Let’s take it a step further and talk about automation and intelligent Cloud management. If I tag instances properly I can automate tasks to control my Cloud based on those tags. For example, maybe I’m a nice guy and don’t make my development team work weekends. I can set up a task to shutdown any instance with “Environment = Development” tag every Friday evening and start again Monday morning. Maybe I want to have an application only online at month end. I can set up another task to schedule when it is online and offline. Tags give us the ability to see what we are paying for and the hooks to control that cost with automation.

I would be remiss if I didn’t point out that tags are an important part of using some great 2nd Watch offerings that help manage your AWS spend. Please check out 2W Insight for more information and how to gain control over and visibility into your cloud spend.

-Keith Homewood, Cloud Architect

Facebooktwittergoogle_pluslinkedinmailrss

An Introduction to CloudFormation

An Introduction to CloudFormation

One of the most powerful services in the AWS collection set is CloudFormation. It provides the ability to programmatically construct and manage a grouping of AWS resources in a predictable way.  With CloudFormation, provisioning of an AWS environment does not have to be done through single CLI commands or by clicking through the console, but can be completed through a JSON (Javascript Object Notation) formatted text file, or CloudFormation template.  With CloudFormation, you can build a few or several AWS resources into an environment automatically.  CloudFormation works with several AWS resource types; from AWS network infrastructure (VPCs, Subnets, Routing Tables, Gateways, and Network ACLs), to compute (EC2 and Auto Scaling), to database (RDS and ElastiCache), to storage (S3) components.  You can see the full list here.

The general JSON structure looks like the following:

CloudFormation1

A template has a total of six main sections; AWSTemplateFormatVersion, Description, Parameters, Mappings, Resources, Outputs.   Of these six template sections only “Resources” is required.  However it is always a good idea to have other sections like Description or Parameters. Each AWS resource has numerous resource type identifiers that are used to extend functionality of the particular resource.

Breaking Down a CloudFormation Template

Here is a simple CloudFormation template provided by AWS.  It creates a single EC2 instance:

CloudFormation2

This template uses the Description, Parameters, Resources, and Outputs template sections.  The Description section is just a short description of what the template does. In this case it says the template will, “Create an EC2 instance running the Amazon Linux 32 bit AMI.”  The next section, the Parameters section is allowing the creation of a string value called KeyPair that can be passed to the stack at time of launch.  During stack launch from the console you would see the following dialogue box where you specify all of the editable parameters for that specific launch of the template, in this case there is only one parameter named KeyPair:

CloudFormation3

Notice how the KeyPair Parameter is available for you to enter a string, as well as the description that was also provided of what you should type in the box, “The EC2 Key Pair to allow SSH access to the instance”.  This would be an existing KeyPair in the us-east-1 region that you would use to access the instance once it’s launched.

Next, in the Resources section, the name “Ec2Instance” is defined as the name of the resource and then given the AWS Resource Type “AWS::EC2::Instance”.  The AWS Resource Type defines the type of AWS resource that the template will be deploying at launch and allows you to configure properties for that particular resource.  In this example only KeyName and ImageID are being used for this AWS resource.  For the AWS Resource type “AWS::EC2::Instance“ there are several additional properties you can use in CloudFormation, you can see the full list here.  Digging deeper we see the KeyName value is a reference to the parameter KeyPair that we defined in the Parameters section of the template, thus allowing the instance that the template creates to use the key pair that we defined at launch.  Next, the ImageId is ami-3b355a52 which is an Amazon Linux 32 bit AMI in the us-east-1 region, and why we have to specify a key that exists in the that region.

Finally, there is an Outputs template section which allows you to return values to the console describing the specific resources that were created. In this example the only output defined is “InstanceID”, which is given both a description, “The InstanceId of the newly created EC2 instance”, and a value, { “Ref” : “Ec2Instance” }, which is a reference to the resource that was created.  As you can see in the picture below, the stack launched successfully and the instance id i-5362512b was created.

CloudFormation4

The Outputs section is especially useful for complex templates because it allows you to summarize in one location all of the pertinent information for your deployed stack.  For example if you deployed dozens of machines in a complex SharePoint farm, you could use the outputs section of the template to just show the public facing endpoint, helping you quickly identify the relevant information to get into the environment.

CloudFormation for Disaster Recovery

The fact that CloudFormation templates construct an AWS environment in a consistent and repeatable fashion make them the perfect tool for Disaster Recovery (DR).  By configuring a CloudFormation template to contain all of you production resources you can deploy the same set of resources in another AWS Availability Zone or another Region entirely.  Thus, if one set of resources became unavailable in a disaster scenario, a quick launch of a CloudFormation template would initialize a whole new stack of production ready components.  Built an environment manually through the console and still want to take advantage of CloudFormation for DR? You can use the CloudFormer tool.  CloudFormer helps you construct a CloudFormation template from existing AWS resources.  You can find more information here.  No matter how you construct your CloudFormation template, the final result will be the same, a complete copy of your AWS environment in the form of JSON formatted document that can be deployed over and over.

Benefits of CloudFormation

The previous example is a very simple illustration of a CloudFormation template on AWS.

Here are some highlights:

  1. With a CloudFormation template you can create identical copies of your resources repeatedly, limiting the complex deployment tasks of sometimes several hundred clicks in the console.
  2. All CloudFormation templates are simple JSON structured files that allow you to easily share them and work with them using your current source control processes and favorite editing tools.
  3. CloudFormation templates can start simple and build over time to allow the most complex environments to be repeatedly deployed.  Thus, making them a great tool for DR.
  4. CloudFormation allows you to customize the AWS resources it deploys through use of Parameters that are editable during runtime of the template. For example if you are deploying an auto scaling group of ec2 instances within a VPC it is possible to have a Parameter that allows the creator to select which size of instance will be used for the creation of the stack.
  5. It can be argued, but the best part about CloudFormation is it’s free!

-Derek Baltazar, Senior Cloud Engineer

Facebooktwittergoogle_pluslinkedinmailrss