1-888-317-7920 info@2ndwatch.com

Benchmarking Amazon Aurora

Amazon Web Services, Microsoft Azure and Google Cloud Platform are the clear leaders in the Cloud Service Provider (CSP) space. The competition between these players drives innovation and results in customers receiving more benefits and better business outcomes over older technologies. The database as a service space is a massive opportunity for customers to gain more in terms of performance, scalability, and high-availability without the overhead or long-term contracts from traditional solutions.

At 2nd Watch we see a lot of customers who want to utilize database as a service to get better performance without the overhead or having all the hassles of traditional database products.  In that, we watch all three Cloud Service Providers very closely, their services and run benchmark comparisons on their products. In our opinion, Amazon Aurora is probably the best-performing database as a service platform that we’ve seen to date, while still being very cost-effective for the performance benefits.

Amazon Aurora is AWS’s fully-managed MySQL-compatible relational database service. Aurora was built to provide commercial-grade performance and availability, while being as easy to use and as cost-effective as an open source engine. Naturally, a lot of people are interested in Aurora’s performance, and a number of people have tried to benchmark it. Benchmarking accurately is not always easy to do, and it’s why you see different results from various benchmarks.  We’ve helped many of our customers benchmark Aurora to understand how it would perform to meet their needs, and in many cases we have migrated, or are actively working on migrating, customers to Aurora.

Recently, Google benchmarked their own Cloud SQL data against Amazon Aurora. Since a number of our customers use Aurora, we were interested in digging in to take a deeper look at their findings.

The chart below shows the benchmark results published by Google.

Graph 1

Original Source: Google. See: https://cloudplatform.googleblog.com/2016/08/Cloud-SQL-Second-Generation-performance-and-feature-deep-dive.html

In their blog post about this comparison, the blog claims that 1) Cloud SQL performs better at low thread counts and 2) customers do not use a lot of threads, so the results at higher thread counts do not matter. Our experience is that the latter claim is not true, especially as we look at our enterprise customers.  So, we questioned our results from previous benchmarking. Therefore, we decided to benchmark things again to see if our results would fall in line with Google’s results. Below you can see an overlay of our data on top of their original graph. Since we do not have the original results from Google’s , an overlay is the best we can do for comparison.

To perform our own benchmarks, we used Terraform to stand up a VPC, our instances of Aurora, and the hosts from which we’d run the sysbench s, all within the same Availability Zone.  We used the same settings and commands as outlined in the Google blog post to against our results.  We shared our results with the Aurora team at AWS, and they confirmed that they saw very similar results when they ed Aurora with sysbench in a configuration similar to ours.

Full data from our s, R scripts, and automation to stand up the infrastructure can be found on github.

Here is a summary of our observations on this study:

1.The data for Aurora’s performance is incorrect

This data does not match the Aurora performance we see when we run the ourselves. Without any special tuning, we saw higher performance results for Aurora. Without the original data from the s Google conducted it is difficult to say why their results differ. All we can say is that, from our ing, Aurora does appear to be more performant and is clearly the database of choice, especially when dealing with higher thread counts.

2nd Watch overlay in purple:

Graph 2

Results from our s on Aurora:

Both studies agree that Aurora has a significant performance advantage above a certain thread count. Google’s shows the previous release of Aurora having an advantage at 16 threads or more (Cloud SQL starts higher but then drops below Aurora). Our shows the la version of Aurora having an advantage at 8 threads, with results between the two databases being fairly comparable at 4 or fewer threads.

2.The study makes a claim that customers don’t need many threads

Google is suggesting that you should only focus on the results with lower thread counts. Their blog post says that Cloud SQL “performed better than Aurora when active thread count is low, as is typical for many web applications.” The claim that low thread counts are typical for web applications (and thus what people should focus on) is inaccurate. There are a large number of AWS customers who run applications using Aurora, and it is very common for them to run with hundreds of threads (and many customers choose to run with thousands). As we mentioned previously, both our study and Google’s study show that Aurora has an advantage at higher thread counts.

There are a number of areas where Amazon improved on the performance of MySQL when developing Aurora. One of them was taking advantage of high thread counts to deliver better performance. With Aurora, customers with higher thread counts will typically see a large performance advantage over MySQL and Cloud SQL.

3.Aurora’s largest instance outperforms Cloud SQL’s largest instance by 3X

Google’s benchmark was done on an r3.4XL, which is only half the size of Aurora’s largest instance (the r3.8XL). We understand that this study used Cloud SQL’s largest machine and that they were trying to compare to a comparable Aurora instance size. But a customer is more likely to run a workload like the one in Google’s benchmark on Aurora’s largest machine, the r3.8XL, because the 8XL’s larger cache would improve performance. We ran the with Aurora’s r3.8XL instance, and the results were more than 3X higher than Cloud SQL’s performance. It was a struggle to fit Aurora’s r3.8XL performance on the same scale.

Aurora on r3.8xlarge (scale adjusted):

Graph 3

A few more things to consider

We’ve already pointed out Aurora’s performance advantage, which is evident in both studies. Aurora has a number of other advantages over Google Cloud SQL. One example is scalability. We showed earlier that Aurora’s largest instance exceeds the performance of Cloud SQL’s largest instance by 3X in this benchmark. Aurora can scale up to handle more storage (64 TB for Aurora vs 10 TB for Cloud SQL). Aurora supports up to 15 low-latency read replicas that have very low replica lag. Aurora replica lag is typically only a few milliseconds, whereas traditional binlog-based replication, which is used by MySQL and Cloud SQL, can result in replica lag on the order of seconds or even minutes. Aurora allows up to 15 replicas that can act as failover targets without impacting the performance of the primary instance, whereas Cloud SQL allows only one failover target. Another advantage is that Aurora has extremely fast crash recovery, as its log-structured storage system does not require the replay of redo logs after a database crash.

Benchmarking Aurora

If you’re interested in benchmarking Aurora, check out AWS’s Aurora benchmarking guide here.

If you’re thinking about benchmarking Aurora yourself, there are two common errors to watch out for. First, make sure you put the database and the client in the same Availability Zone or you will pay a latency and throughput penalty. This is what customers actually do when they use Aurora in production. Second, make sure that the client instance driving traffic to Aurora has enhanced networking enabled. The benchmarking guide mentioned above has instructions for how to do that.

Making good use of benchmarking data is tricky. You’re trying to build or migrate your application and you want to understand how your new database is likely to behave, now and in the future. There’s no magic benchmark that will exactly match your production workload, but you need to make a decision anyway. What should you do?

Here’s some general advice we try to follow for our own projects as well.

  1. Plan for success by ing at scale: The last thing you want to do is make design choices that hurt you when your application starts to really take off. That’s when you want to celebrate with your team and build the next great feature, not attempt an emergency database migration. Think about how your workload might change as your application grows (higher thread counts, more data, higher TPS, more tables…) and build that into your s.
  2. Choose benchmarking s that align to your application: Do you expect your data set to fit in memory or will your database be disk-bound? Will your traffic pattern be spiky? Write-heavy? You can find a large number of benchmarks for any major technology. Try to identify a set of s that are as close as possible to your real-world application. Don’t just accept the first benchmarking study you find. If you can re-run your actual production workload as a , even better.
  3. Know that your benchmarks are not the real world and plan for that: The best benchmarking study you run will be, at best, slightly wrong (not the same as the real world). Make a note of the assumptions you made in your and keep an eye on your database in production. Watch for signs that your production workload is moving into uned areas and adjust your s accordingly.
  4. Get help: The best way to get expert advice is to see an expert. Cloud technology is complex and not something you should seek guidance on from partners who are “generalists.” To Aurora at your company, visit us at www.2ndwatch.com.


-Chris Nolan, Director of Product & Lars Cromley, Sr. Product Manager


AWS a Leader Again in 2016 Gartner Magic Quadrant

Gartner’s 2016 Magic Quadrant for Cloud Infrastructure as a Service, Worldwide was released today, evaluating 10 different vendors for ‘completeness of vision’ and ‘ability to execute.’ Although AWS continues to face more competition, we know it remains a market share leader in the industry. Gartner has placed AWS as having both the furthest completeness of vision and the highest ability to execute.

Because the market for cloud IaaS is in a state of upheaval, with many service providers shifting their strategies after failing to gain enough market traction, 2nd Watch recommends utilizing Gartner’s Magic Quadrant to help eliminate the confusion around the various providers in the Infrastructure as a Service sector.

Access the Gartner Report

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

-Nicole Maus, Marketing Manager


And the top AWS products of Q1 2016 are…

AWS is an innovation lab. The world’s top cloud provider releases hundreds of updates and dozens of major services every year. So, which products are companies loving right now?

We analyzed data from our customers, across a combined 100,000+ instances running monthly, for Q1 of 2016. The most popular AWS products, represented by the percentage of 2nd Watch customers deploying them in the first quarter, include Amazon S3 for storage and Data Transfer (100% each), EC2 (99%), SNS or Simple Notification Service (89%) and Key Management Service for encryption (87%). These services are standard in most AWS deployments, and have been consistent in the last year or so – no surprises here.

Perhaps less predictable was the use of other AWS products, such as Redshift, the data warehouse service introduced in 2012 as a low-cost alternative to systems from HP, Oracle and IBM. The fact that 17% of our customers are using Redshift demonstrates how quickly innovative cloud technology can carve a strong position in a legacy software market. Enterprises are starting to move away from legacy systems to Redshift because it can handle massive quantities of data with exceptional response times.

Other relatively new AWS products making rapid progress with AWS users include the high-performing NoSQL database service Dynamo DB (27%), Lambda, an automated compute management platform (21%) and Workspaces, a secure virtual desktop service (19%).

Just three years ago, enterprises were primarily using the core compute and storage services on AWS. As companies become more comfortable moving business-critical IT assets into the cloud, they’re more likely to leverage the broader AWS portfolio. We expect growth in areas such as database, desktop and management tools to continue in the coming months.

Download the Top 30 AWS Products infographic to find out which others made the list.

-Jeff Aden, EVP Strategic Business Development & Marketing


Database Migration Service for RDS

For database administrators and engineers, migrating a database can be a major headache. It’s such a headache that it actually prohibits some teams from migrating to AWS’ Relational Database Service (RDS), even though doing so would save them time and money in the long run.

Imagine you’re a DBA for Small Business Y and you want to manage your data in RDS, but you have three terabytes of data with a ton of tables, foreign keys and many dependencies. Back in the day, migrating your SQL Server database to RDS might have looked something like this:

  1. Coordinate with Product Leads to find a time when your business can handle up to a 24-hour outage of source database.
  2. Dump all the existing data into a backup.
  3. Restore the data on an intermediary EC2 SQL Server instance.
  4. Connect to the new RDS instance.
  5. Generate metadata script from the source or intermediary instance.
  6. Execute metadata script on target RDS instance.
  7. Migrate the data to the RDS instance using SQL Server’s Import tool.
  8. Optional: Encounter complications such as import failures, loss of metadata integrity, loss of data, and extremely slow import speeds.
  9. Cry a lot and then sleep for days.

Enter AWS Database Migration Service. This new tool from AWS allows DBAs to complete migrations to RDS, or even to a database instance on EC2, with minimal steps and minimal downtime to the source database. What does that mean for you? No 2AM migrations and no tears.

Migrations with AWS DMS have three simple steps:

  1. Provision a replication instance
  2. Define source and target endpoints
  3. Create one or more tasks to migrate data between source and target

The service automates table-by-table migration into your target database without having to use any backups, dump files, or manually administering an intermediary database server. In addition, you can use the service to continue replicating changes from the source environment until you are ready to cutover. That means your application’s downtime will be just a few minutes instead of several hours or even days.

Another great feature of DMS is that you can watch the progress of each table’s migration in the AWS console. The dashboard allows users to see, in real time, how the migration is going and see exactly where any breakages occur.

Database Migration Svc 1


Database Migration Svc 2

If you are planning to use the AWS Database Migration Service soon, here are a few tips to streamline your process:

  • In your target instance, set up your table structure prior to migrating the data. The service will not automatically set up your foreign keys and indexes. Set up your metadata from AWS schema conversion tool (or simple mysqldump). Make sure to use truncate option during schema import so that the tables you create aren’t wiped.
  • If you think you may have extra-large LOBs, use Full LOB Mode to avoid data truncation.
  • Additional best practices for using DMS can be found here.


-Trish Clark, Oracle DBA


Enter AWS Lambda

AWS has managed to transform the traditional datacenter model into a feature-rich platform and has been constantly adding new services to meet business and consumer needs. As virtualization has changed the way infrastructure is now built and managed, the ‘serverless’ execution model has become a viable method of reducing costs and simplifying management. A few years ago, the infrastructure required to host a typical application or service required the setup and management of physical hardware, operating systems and application code. AWS’ offerings have grown to include services such as RDS, SES, DynamoDB and ElastiCache which provide a subset of functionality without the requirement of having to manage the entire underlying infrastructure on which those services actually run.

Enter AWS Lambda.

Lambda is a serverless compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security.

In a nutshell, Lambda provides a service that executes custom code without having to manage the underlying infrastructure on which that code is executed. The administration of the underlying compute resources, including server and operating system maintenance, capacity provisioning, automatic scaling, code monitoring, logging, and code and security patch deployment are eliminated. With AWS Lambda, you pay only for what you use, and are charged based on the number of requests for your functions and the time your code executes. This allows you to eliminate the overhead of paying for instances (by the hour or reserved) and their administration. Why build an entire house if all you need is a kitchen so you can cook dinner? In addition, the service also automatically scales to meet capacity requirements. Again, less complexity and overhead than managing EC2 Auto Scale Groups.

Lambda’s compute service currently supports Node.js (JavaScript), Python, and Java (Java 8 compatible). Your code can include existing libraries, even native ones. Code is executed as what is referred to as a Function.

Here’s AWS’ Jeff Barr’s simple description of the service and how it works:

You upload your code and then specify context information to AWS Lambda to create a function. The context information specifies the execution environment (language, memory requirements, a timeout period, and IAM role) and also points to the function you’d like to invoke within your code. The code and the metadata are durably stored in AWS and can later be referred to by name or by ARN (Amazon Resource Name). You can also include any necessary third-party libraries in the upload (which takes the form of a single ZIP file per function).

After uploading, you associate your function with specific AWS resources (a particular S3 bucket, DynamoDB table, or Kinesis stream). Lambda will then arrange to route events (generally signifying that the resource has changed) to your function.

When a resource changes, Lambda will execute any functions that are associated with it. It will launch and manage compute resources as needed in order to keep up with incoming requests. You don’t need to worry about this; Lambda will manage the resources for you and will shut them down if they are no longer needed.

Lambda Functions can be invoked by triggers from changes in state or data from services such as S3, DynamoDB, Kinesis, SNS and CloudTrail, after which, the output can then be sent back to those same services (though it does not have to be). It handles listening, polling, queuing and auto-scaling and spins up as many workers as needed match the rate change of source data.

A few common use cases include:

  • S3 + Lambda (Dynamic data ingestion) – Image re-sizing, Video Transcoding, Indexing, Log Processing
  • Direct Call + Lambda (Serverless backend) – Microservices, Mobile backends, IoT backends
  • Kinesis + Lambda (Live Stream Processing) – Transaction Processing, Stream analysis, Telemetry and Metering
  • SNS + Lambda (Custom Messages) – Automating alarm responses, IT Auditing, Text to Email Push

Additionally, data can be sent in parallel to separate Functions to decrease the amount of time required for data that must be processed or manipulated multiple times. This could theoretically be used to perform real-time analytics and data aggregation from a source such as Kinesis.

Function overview

  • Memory is specified ranging from 128MB to 1GB, in 64MB increments. Disk, network and compute resources are provisioned based on the memory footprint. Lambda tells you how much memory is used, so this setting can be tuned.
  • They can be invoked on-demand via the CLI and AWS Console, or subscribed to one or multiple event sources (e.g. S3, SNS). And you can reuse the same Function for those event sources.
  • Granular permissions can be applied via IAM such as IAM Roles. At a minimum, logging to CloudWatch is recommended.
  • Limits to resource allocation such as 512MB /tmp space, 1024 file descriptors and 50MB deployment package size can be found at http://docs.aws.amazon.com/lambda/la/dg/limits.html.
  • Multiple deployment options exist including direct authoring via the AWS Console, packaging code as a zip, and 3rd party plugins (Grunt, Jenkins).
  • Stateless data means depending on another service such as S3 or DynamoDB to retain persistence.
  • Monitoring and debugging can be accomplished using the Console Dashboard to view CloudWatch metrics such as requests, errors, latency and throttling.

Invoking Lambda functions can be achieved using Push or Pull methods. In the event of a Push from S3 or SNS, retries occur automatically 3 times and is unordered. One event equals one Function invocation. Pull, on the other hand (Kinesis & DynamoDB), is ordered and will retry indefinitely until data expires. Resource policies (used in the Push model) can be defined per Function and allow for cross-account access. IAM roles (used for Pull), can be used to derive permission from execution role to read data from a particular stream.


Lambda uses a fine-grained pricing model based on the number of requests made AND the execution time of those requests. Each month, the first 1 million requests are free with a $0.20 charge per 1 million requests thereafter. Duration is calculated from the time your code begins executing until it returns or otherwise terminates, rounded up to the nearest 100ms and takes into account the amount of memory allocated to a function. The execution cost is $0.00001667 for every GB-second used.

Additional details regarding the service can be found https://aws.amazon.com/lambda/. If you need help getting started with the service, contact us at 2nd Watch.

-Ryan Manikowski, Cloud Consultant




AWS Launches Visual App for CloudFormation

New visualization tool is the first native visualization for AWS infrastructure, and a groundbreaking development in the adoption of infrastructure as a code.

Cloud developers and architects use AWS CloudFormation to design, launch and update an AWS application or service architecture stack for repeatable workloads. CloudFormation provides templates for creating an entire environment, such as a website, using the JSON scripting language. Developers don’t need to figure out the order for provisioning AWS services or worry about the dependencies, as the free tool handles all the behind the scene configurations.

This is a huge help when you have workloads that must launch over and over again, saving time on provisioning and configuration. You can actually launch “with the click of a button.” On the other hand, CloudFormation files typically consist of thousands of lines of code, which doesn’t make them easy to modify or share.

Now, with the new AWS CloudFormation Designer, customers can view the details behind an AWS environment through a simpler, graphical view. You don’t need JSON expertise to collaborate on design and planning. The visual drag-and-drop interface is remarkably easy to use, compared with how cloud developers have been working thus far. Instead of writing several lines of code, you can draw a line on the screen connecting one resource to another.

The big picture

AWS CloudFormation Designer will expand the universe of IT people who can write AWS scripts and manage workloads, since they won’t need to know JSON. That means more hands on deck for new projects. After you create the visualization in the tool, you can launch the environment right then and there. We’ve seen that CloudFormation Designer reduces the time to create and launch a new workload from hours to minutes.

What’s also intriguing about this new feature is that it further affirms the infrastructure-as-code mindset. Consider the impact that tools like Visual Studio, Borland and IDEs have had on Microsoft .NET and Java developers. Developers can write code faster and reduce errors. Similarly, AWS has in effect created an interactive development environment (IDE) type tool for the cloud. This is a radical departure for the public cloud leader; AWS is known for its boxes and pipes, not sophisticated visual tools.

AWS CloudFormation Designer is a refreshing development for those who believe public cloud infrastructure will take over the world. One of the challenges with cloud computing adoption is that infrastructure as a service remains a new language to many corporate IT departments. AWS has made a smart move here by introducing more transparency to its platform, and opening the door for IT generalists to get involved. We’re excited that our customers and other companies will benefit from this new view of AWS.

-Kris Bliesner, Founder and CTO