Every enterprise knows by now that it can save money by simply lifting and shifting workloads to the cloud, but many are missing the larger opportunity to also make money by moving. While quick costs savings are good for the bottom line, they do little to move the top line numbers. To achieve both savings and earnings, corporate thinking about technologies must change in order to enable faster processes leveraged enterprise-wide.
In this AWS re:Invent 2016 breakout session we explored multiple customer success stories where the customers have evolved from leveraging basic compute and storage products (EC2 and S3) to integrating new services into operations by leveraging Lambda, DynamoDB, CodeDeploy, etc. Once this is achieved, enterprises are enabled to manage and deploy code rapidly in a programmatic and elastic secure network, ensuring governance and security standards across the globe. We also looked at the migration process trusted by hundreds of clients as well as how to cope with the process and people components that are so important to enable agility, while focusing heavily on the technology.
Dive deep into the technology that allows the world’s largest beverage manufacturer to manage hundreds of AWS Accounts, hundreds of workloads, thousands of instances, and hundreds of business partners around the globe. The company’s Configuration Management System has Puppet at the core and relies on over a dozen core and emerging AWS products across accounts, availability zones and regions. This complex and globally-available system ensures all of company’s workloads in AWS meet corporate policies but also allows for rapid scale of both consumer and enterprise workloads.
After the deluge of announcements during Andy Jassy’s Wednesday keynote, I had a hard time imagining what else AWS could possibly have left to announce yesterday. Of course, in typical AWS fashion, they had a lot more to talk about.
Werner Vogel, Amazon’s CTO, laid out a strong case that developers, data analysts, and basically just about anyone involved with data spend 80 percent of their time preparing data to become usable — and not always successfully — while only spending 20% of their time on the actually analysis and use of that information. He laid out a vision for a new, modern data architecture – one that would flip that equation and therefore transform the way with which we can generate value and insight from all our data sources. The transformer theme, woven throughout Vogel’s talk (and even his shirt), was so pronounced that one wonders why the new AWS Snowmobile didn’t have an Optimus Prime paint job.
In pursuit of enabling this transformation, the Keynote showcased a large number of new and enhanced services, most centered around removing a lot of the scut work that reduces development speed, efficiency, and agility.
AWS OpsWorks for Chef: A fully managed Chef Automation environment that helps take a lot of the work out of continuous deployment.
Amazon EC2 Systems Manager: A suite of tools for task automation, package installation, resource configuration, and patching on Amazon EC2.
AWS Codebuild: A fully managed and extensible build service for compiling source code and running unit s. Codebuild integrates with a wide array of AWS services, and helps make CI/CD pipelines more efficient.
Personal Health Dashboard: Allowing developers to gain visibility into service health issues that may be affecting their application.
Amazon X-Ray: Another debuting tool, X-Ray lets developers analyze, visualize and debug distributed applications and identify performance bottlenecks.
AWS Shield: A new AWS security tool designed to provide layer 3/4 DDoS protection to web applications.
AWS Batch: Batch offers fully-managed, highly scalable batch processing without having to install batch processing software, manage servers, or worry about the finer points of job scheduling.
AWS Glue : A fully-managed data catalog and ETL service that makes it easy to transfer data among data stores while also simplifying associated tasks such as data discovery, conversion, and mapping.
Amazon Pinpoint: A new analytics tool to improve the behavioral analysis and engagement with mobile customers.
AWS Step Functions: Lets developers organize the components of distributed applications using visual workflows. Users can step through functions at scale, improving debugging.
Blox: A collection of open source projects for container management and orchestration.
Lambda@Edge: While just a preview at this point, this service enables Lambda functions at AWS edge locations, as well as execution in response to CloudFront events.
Sessions and Events
As usual, the conference agenda was stuffed to the gills with interesting and useful events, session, and demos. The meat of AWS, of course, is the wide array of breakout sessions. I was able to attend a few, three of which were particularly strong.
Lee Atchison from New Relic gave a particularly interesting talk on cloud monitoring and how best to architect applications and infrastructure to ensure they are fully measurable. This is challenging under any circumstances, but particularly so as applications and their underlying resources become more dynamic, more ephemeral. Gannet News discussed how they were able to transform their systems in this regard, offering a lot of insight into how to create and manage dynamic IT.
More architectural advice, this time with a focus on security, came from an AWS-led session on architecting end-to-end security in the enterprise, while a related session, also AWS-led, explored how to best automate security event response in such an architecture. This last is critical to have scalability and predictability when remediating security issues. In combination, these two sessions helped lay out an approach for highly dynamic, manageable, and secure systems.
The expo hall was very heavily-trafficked, with booths offering monitoring solutions and security analysis and governance being especially busy. IoT was also a topic of great interest during the conference, with several intriguing vendor offerings.
At this morning’s AWS re:Invent keynote, AWS shared quite a mountain of information, and a toolbox of new services, all based around helping companies change their businesses and the way they look at technology. Transformation was the keyword for this presentation, and it was apparent in the tools and tone taken throughout the whole two and a half hours. The focus was on providing the tools to the “Transformers” (Highlighted by Vogel’s Autobot T-shirt), and enabling them to do amazing things for their customers. Vogel’s keynote was less about infrastructure, more about the software and how to get it into the hands of your customers, and how the toolbox that AWS continues to expand can help. It’s not entirely about AWS though… it starts with their customers.
AWS: To Be the Most Customer Centric IT Company on Earth
There’s a large drive from all the teams at AWS to focus on the needs of their customers (that’s you by the way). In fact, this couldn’t be more evident than with their new offering called AWS Blox, an open source scheduler for ECS that’ll be co-developed with the community. This can also be seen in their 5 customer centric objectives:
Protect the customers at all times.
Listen closely to customers and act.
Give customers choice.
Work backwards from the customer.
Help customers transform.
This led nicely into Jeff Lawson’s (CEO / Chairman – Twilio) presentation which revolved around software development. The two things to take away from this were a couple of quotes: 1. “Building software is a mindset, not a skillset,” which speaks immeasurably to the idea of the enveloping purpose of software in the first place. Software drives products to customers. And 2. “Companies that win are companies that ship software.”
How can we help you be a Transformer?
There are a plethora of modern day processes revolving around Agile practices, which involve feature deployment speed to your customers. The big, main point here is that Amazon really wants to take as much of the waste off of their customers’ shoulders as possible and manage it for them. This is one of the fundamental principals in lean manufacturing and Agile development processes. Cut waste, so your people can concentrate on what’s important to your customer – Providing stellar products and features.
To that end, AWS already provides everything you’ll need as far as infrastructure is concerned. Need a thousand instances for a load ? Spin them up, run your , then tear them down, and only pay for that hour you had them up. That’s the bread and butter. Where AWS is moving now is to help that development pipeline and to provide the tools to do it.
First and foremost, they’ve updated their Well Architected Framework (along with all the underlying documentation) to include a 5th pillar:
Operational Excellence (This is where Automation and CI/CD pipelines come into play.)
Transforming Operational Excellence
Automation is the name of the game here. The existing tools have gotten some updates, and there are some new ones to add to your armory as well.
AWS CloudFormation has seen a ton of updates this past year including role-based stack creation, failure recovery, resource schemas and last but by far not least, yaml support! Configuration management (in the form of Chef) has gotten a BIG boost in their new AWS Opsworks For Chef Automate, a fully managed chef server. Oh, and managing system level patching and resource configuration? They’ve got that covered as well with the Amazon EC2 Systems Manager. The Biggest changes come to help your CI/CD pipeline. The new AWS CodeBuild will build and your projects and fills out the pipeline toolset (between CodeCommit and CodeDeploy). What about insight into your application? The fantastic looking X-Ray will allow insight into your applications on a very deep level, with a smart looking UI to boot. Another nice looking UI of a tool to handle managing events from your infrastructure is AWS Personal Health Dashboard. This tool will help you manage responses to your events, and can be tied into Lambda for automation.
Security is number one with AWS, so it’s no surprise that they’re offering two new tools to help protect against the common DDOS attack. The first, AWS Shield will help protect against some of the more common DDOS attack vectors. The best thing about it? Everyone gets it FOR FREE! You use AWS, you get AWS Shield. That simple. AWS Shield Advanced is for more complex attacks and is a paid service that you can opt in for if you feel the need.
Transforming your Data
Amazon’s cloud offering levels the playing field when it comes to resource procurement. Small companies can now compete with the big ones since they draw from the same pool and have the same tools available to them (regardless of size). So what’s your competitive differentiator? Data. That’s why another focus of this past year has been on Big Data.
AWS already has a lot going for it with data analytics, from ingestion tools like Kinesis and Snowball to processing with EMR, there just seemed to be one thing missing: AWS Glue. AWS Glue pulls together all the components of Modern Data Warehouses into a comprehensive architecture for data analytics. From data ingestion to data quality, source data preservation to orchestration and job scheduling, it looks like AWS Glue will manage it all. Also on the processing end, the new AWS Batch tool will manage batch processing at any scale.
Transforming your Application Architecture
Amazon now provides 3 different architectures and payment styles when it comes to application development (or deployment if you look at it that way) – Virtualization, which is already quite robust in their compute ecosystem; Containers, which have an ever maturing product in ECS; and Serverless, which is handled quite well through services like AWS Lambda. Virtualization didn’t get a particular mention here, but Containerization did. Blox was already mentioned above, but there was also a “coming soon” drop here as well. Looks like we’ll be seeing some kind of task placement engine in the near future.
Next up were new offerings around Lambda. The first, and one that will surely broaden the adoption of serverless architectures, is the inclusion of the C# language into the list of supported languages. To cut back on possible latency issues, you can now run Lambda functions at CloudFront locations using the new AWS Lambda@Edge. To help coordinate all the components of your distributed applications, you now have AWS Step Functions. This tool will allow you to coordinate all your bits and pieces using a visual workflow.
There’s a lot of potential for transforming your business here.
Like always, AWS doesn’t force you to use any particular tool or service, but they have a lot of what you need to develop products and features the right way. They’ve made some serious strides to pull as much of the wasted, non-customer centric work away from your teams, and give them back that time to push more value to your customers. Amazon doesn’t yet approach the organizational / process side of the equation, so that will still fall to the customer. Once you figure it out though, it looks like AWS is positioned, and will continue to position itself, to help you and your teams make that transformation a reality.
AWS re:Invent 2016 day 2 is in the books. Attendees took part in AWS and Partner-led technical sessions, bootcamps, and GameDay and had the opportunity to interact with a wide range of Consulting and Technology Partners at reinvent:Central (the Partner & Solutions Expo).
Things started off with the re:Invent Keynote by AWS CEO Andy Jassy, and per the usual, he did not disappoint. We learned about the Four Superpowers that AWS gives organizations… Supersonic Speed, X-Ray Vision, Immortality, and Flight. and we heard success stories around this from several major corporations. Speakers from GE, Motorola, Workday, FINRA and McDonalds joined the Keynote and told their success stories as they relate to AWS and these Four Superpowers. We also heard from Patrick Gelsinger, CEO of VMware, and Jassy about the new partnership between AWS and VMware. They discussed the alliance and the strategic and technical benefits that it will provide their customers. Gelsinger notably quipped: “Your CIOs capital budget for DR should be zero. DR should entirely be in the cloud.”
New Announcements The Summit Keynote showcased a number of new and expanded services to the AWS lineup:
New C5 EC2 Instance Family: Next Generation Compute Optimized Instances with Intel Skylake processors
New I3 EC2 Instance Family: Next Generation I/O Intensive Instances
New R4 EC2 Instance Family: Next Generation Memory Optimized Instances
New F1 EC2 Instance Family: Compute instances with a Field Programmable Gate Arrays (FPGAs)
Updates to the T2 EC2 Instance Family: Low-Cost Instances with Burstable Performance
xlarge with 16GiB (2x more) memory and 2 vCPU
2xlarge with 32GiB Memory (4x more) and 2vCPU
Addition of Elastic GPUs to all EC2 instance types
PostgreSQL support for RDS Aurora
Amazon Athena: Provides serverless interactive queries for data stored in S3 using Standard SQL
Amazon Rekognition: Image recognition service that enables applications to easily recognize, search, and understand image details
Amazon Polly: Text to speech service with 47 voices and 27 languages
Amazon LEX: Natural language understanding and automatic speech recognition service (It’s what’s inside Alexa.)
Jassy unveiled the four new EC2 instance families – three of which are the la iterations on previous generations – listed above as well as the expansion of the T2 line. and types. The completely new EC2 instance family was the F1 class. F1 instances offer programmable hardware via an FPGA which allows for writing custom code that executes directly on the FPGA speeding up the processing time for discrete tasks. We also learned that AWS is adding Elastic GPUs as an option (handled much like an EBS volume) that will be available to attach to any EC2 instance to boost graphics processing performance.
Amazon Athena was announced today and should be a welcomed addition for anyone looking to turn their S3 data into a data-warehouse/lake instantaneously without incurring the costs associated with a Redshift cluster, infrastructure, or other data-warehousing platform. Athena provides serverless interactive queries for data stored in S3 using Standard SQL. This is a truly awesome new service for building a data-warehouse inexpensively.
Perhaps one of the biggest announcements of the day came as it was announced that AWS will be releasing PostgreSQL support for Aurora. This will definitely help Aurora continue to reshape the face of how enterprise class databases are deployed in the cloud. Speaking of Aurora reshaping database deployment in the cloud, Jassy mentioned that since its launch Aurora is the fasting-growing service AWS has ever released with 14k+ migrations to date.
The launch of the Amazon AI platform was another major announcement made yesterday. Leveraging the IP and tooling they have developed internally, AWS will be releasing three new services under this new AI platform aimed at providing AI services to the masses. The first is Amazon Rekognition and provides an image detection and recognition engine. The next is Amazon Polly, which provides a text-to-speech engine (in 47 languages with 24 voices). The last is Amazon LEX, which is a natural language understanding and automatic speech recognition engine, which operates at the core of the Amazon Alexa platform. This is just another great example of Amazon providing internal tools and services they have developed and bringing them to a global marketplace due to customer demand.
There was a flurry of activity on the re:Invent Central expo floor, and traffic to the 2nd Watch booth was no exception. New customers were met, partnerships were discussed and all kinds of cloud expertise was disseminated to the masses.
Sessions and Events The Agenda was chock-full of interesting and useful events, session, and demos in addition to the Keynote Address. Events like the Alexa Skill Con, Certification Lounge provided an all-day option for honing skills and gaining AWS certification. Some, like the live demo “Creating an AWS Data Lake on S3,” highlighted the newly-announced AWS services and features. There were also several physical fun activities over at The Park, like Hangry Humans, Mini golf, Mechanical Bull Riding and a Batting cage for attendees to get outside for some fresh air and remain active.
After hours kicked off with the re:Invent 2016 Pub Crawl featuring a number of great restaurants and pubs. The night was finished off with a bang at the annual 2nd Watch re:Invent After Party.
Come see us today at Booth #825 for your free limited edition t-shirt and a chance to win a Segway!
-Ryan Kennedy, Principal Cloud Automation Architect
What a difference a year can make at AWS. In Andy Jassy’s AWS re:Invent 2015 Keynote address, which lasted just shy of 2 hours, he gave us some pretty stunning financial growth statistics and detailed the seven freedoms AWS provides its customers including; the freedom to build unfettered, to get the real value from your data, to get data into or out of the cloud easily, freedom from bad database relationships, freedom to migrate, freedom to secure your cake and eat it to, and freedom to say yes.
This year, in a presentation that lasted almost two and a half hours, Andy updated us on key corporate and re:Invent statistics, discussed the “4 Superpowers” AWS endows upon their customers and, along the way, introduced many significant new products and features that are sure to change the game moving forward. Here are the key things you can take away from this presentation:
AWS continues its exponential growth and commitment to its customers
The participant headcount at re:Invent this year increased to over 32,000 (up from 19,000 in 2015), Live Stream viewership increased from 38,000 in 2015 to over 50,000 and breakout session increased from 248 to over 400 this year. In that same time AWS has grown its business from a mere $7B run rate to over $13B and is by far the fas growing large scale technology company in the world. According to Gartner, AWS has several times the compute capacity of the next 14 cloud providers combined. With their continued investment in new features and technologies it is safe to say this lead will remain well into the future.
IT Professionals Need Even More Compute Options
Agility and speed are two of the main drivers compelling companies to move to the cloud (along with cost savings of course). AWS provides the most comprehensive line of products and deep functionality which enables companies to leverage the cloud to quickly adopt the cloud and speed new products and services to market. Andy emphasized AWS core strategy of having more than one data center in each region to provide customers the resiliency they require and that they continue to launch new products and add new features to existing products in order to make the cloud even more valuable.
AWS has more compute products than anyone else in the market but that is not stopping them from expanding their offering. Today AWS announced the following four new EC2 instance families:
T2 Family: Low-Cost Instances with Burstable Performance
xlarge with 16GiB (2x more) memory and 2 vCPU
2xlarge with 32GiB Memory (4x more) and 2vCPU
New R4 Family: Next Generation Memory Optimized Instances
New I3 Family: Next Generation I/O Intensive Instances
3 million IOPs (9x more) 4.88GiB (2x more) 15 2TB MVMe SSD (2.3x more) and 64 vCPUs (2x more)
New C5 Family: Next Generation Compute Optimized Instances with Intel Skylake processors
72 vCPUs (2x more), Skylake processor (2x greater performance), 12Gbps to EBS (3X greater throughput) and 144GiB memory (2.4x more)
New F1 family (Developer Preview): Instances with access to a Field Programmable Gate Array (FPGA) which can be utilized to more quickly and efficiently execute code for application specific workloads like seismic data, genomics, and financial analysis to name just a few. With the new F1 instances users can now run custom logic on EC2 bringing custom hardware acceleration to anyone.
In addition to these new compute products AWS announced Elastic GPUs for EC2 in sizes from 1 GiB up to 8 GiBs to allow you to attach and use graphics GPUs with an EC2 instance in the same way as an EBS Volumes. Another great new compute product announced today was Lightsail, which is in effect a Virtual Private Sever (VPS) made easy. Lightsail truly revolutionizes VPS by making the deployment a simple three step process for as little as $5 per month. Simply select your image, select your size, and pick a name and AWS does the rest (Launch VM, Attach SSD Storage, Manage IAM, Create Security Groups, setup DNS and create a Static IP) in the background. Now setting up a VPS is about as simple as it can get.
To quote Andy, “We love Ourselves Some Compute”, and it really shows with these new products.
Companies Need Better Analytics, and AWS Can Deliver
The ability to understand your customers and business grows more critical every day. AWS has a broad set of products including Amazon EMR, Elasticsearch, Kinesis, Redshift, QuickSite and Machine Learning. To compliment this comprehensive line of products AWS announced the launch of Amazon Athena. Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. This enables companies to query data without having to move data out of S3 or set up any infrastructure. You get fast results and pay only for the queries you run.
Artificial Intelligence and Deep Learning are clearly advancing how computers are solving intuitive problems. The challenge with AI and Deep Learning has always been that it takes very skilled team to actually make use of these technologies. AWS is working to bring the power of AI and Deep Learning to all developers through the release of the Amazon AI Services: Amazon Rekognition, Amazon Polly, and Amazon LEX.
Amazon Rekognition is an image recognition service that enables applications to easily recognize, search, and understand image details.
Amazon Polly is a new text to speech service with 47 voices and 27 languages.
Amazon LEX (It’s what’s inside Alexa) is a natural language understanding and automatic speech recognition service powered by deep learning which enable programmers to build natural conversational interactions in voice and text.
It’s hard for Enterprises to Stay Relevant and Grow
Only 12 of the companies in the first Fortune 500 list are still there today. To remain relevant and continue growing, enterprises need to be able to quickly capitalize on new technologies. The ones that do not will most likely face extinction in the future. Andy highlighted Workday, GE, Motorola, FINRA, and McDonald as companies who have used the cloud to either build or transform their business. Workday is a $1.5B company built from the ground up in the cloud. GE has moved 9,000 applications to AWS. Motorola is reinventing themselves as a communications platform using AWS services. FINRA is a financial market surveillance company using big data analytics and elastic compute at a massive scale to surveil, analyze, and protect financial markets for the SEC. FINRA chose AWS over traditional data centers not only for the cost savings, tooling, and elastic capabilities they provide but also sited security that is better than traditional private data-centers as being a major factor. McDonalds POS system with two hundred thousand registers is now running on AWS. Each of these companies can now bring new products and services to market faster and can easily scale to meet future demands.
Companies Need to Move Away from Customer Hostile Databases
We have all heard or experienced horror stories of old world databases. With their proprietary designs, lock-in pricing strategies, and punitive licensing agreements these databases have been at best a challenge for many customers. AWS has built a strategy to make open databases easier, more reliable, and faster to use with offerings that include MySQL, PostgreSQL, MariaDB, and Aurora. Aurora provides the speed and availability of commercial databases with the cost-effectiveness of open source and has become their fas growing service ever (with 14k+ migrations to date). And as of today Aurora now includes PostgreSQL support. The combination of these products make it much easier to move away from old world databases.
Choosing between On Premise to Cloud is No Longer a Binary Decision
AWS is making it easier to deploy hybrid IT infrastructure. Six weeks ago they announced a partnership with VMware to enable companies to run the same VMware software on AWS that they run in their data centers. This allows companies to use their existing VMware tools and seamlessly migrate workloads to and from the cloud greatly increasing flexibility while using the tools they are familiar with.
IoT is About to Change in Dramatic Fashion
IoT devices typically have very limited resources on board. To process information, the data has to be transported back to an on-premise or cloud datacenter. This increases latency and bandwidth costs while exposing customers to potential connectivity disruptions. To address these issues, AWS announced Greengrass which embeds Lambda compute (and other AWS services) in connective devices to provide local compute, messaging, and data caching. Soon Greengrass will be available on all Intel and Qualcomm IoT chips as well as Canonical and Annapurnalabs software.
Your Gonna Need a Bigger Truck
Last year AWS launched Snowball a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of the AWS cloud. This product has been a huge success and after listening to users, they are expanding the service. Today AWS announce AWS Snowball Edge providing 100Tb of storage, on-board computing and clustering capabilities, an S3 endpoint and Greengrass inside.
But what about exabyte-scale data?… Believe it or not, enterprises are now faced with moving exabytes of data from on-premise storage to AWS. Moving that much data to the cloud using a 10Gbps dedicated connection would take 26 years. What to do? Use the newly released AWS Snowmobile! Snowmobile is a dedicated semi truck tractor trailer (yes they even drove one on stage) for data transfer. This 45 foot long rugged container and truck can be connected via fiber directly to your datacenter to collect your data then transported to AWS for upload 100 petabytes at a time taking an exabyte migration from 26 years down to 6 months.
AWS is not only the clear leader in the cloud space, but they continue to listen to customers and adapt their product portfolio to meet the ever changing needs of the business world. With up to a thousand new products and features launched each year, you can be sure they will continue to adapt and remain the leader in this space for many decades to come.