The exponential growth of big data is pushing companies to process massive amounts of information as quickly as possible, which is often times not realistic, practical or down right just not achievable on standard CPI’s. In a nutshell, High Performance Computing (HPC) allows you to scale performance to process and report on the data quicker and can be the solution to many of your big data problems.
However, this still relies on your cluster capabilities. By using AWS for your HPC needs, you no longer have to worry about designing and adjusting your job to meet the capabilities of your cluster. Instead, you can quickly design and change your cluster to meet the needs of your jobs. There are several tools and services available to help you do this, like the AWS Marketplace, AWS API’s, or AWS CloudFormation Templates.
Today, I’d like to focus on one aspect of running an HPC cluster in AWS that people tend to forget about – placement groups.
Placement groups are a logical grouping of instances in a single availability zone. This allows you to take full advantage of a low-latency 10 GB network, which in turn will allow you to be able to transfer up to 4TB of data per hour between nodes. However, because of the low-latency 10 GB network, the placement groups cannot span to multiple availability zones. This may scare some people away from using them, but it shouldn’t. You can create multiple placement groups in different availability zones as a work-around, and with enhanced networking you can also still connect between the different HPC’s.
One of the grea benefits of AWS HPC is that you can run your High Performance Computing clusters with no up-front costs and scale out to hundreds of thousands of cores within minutes to meet your computing needs. Learn more about Big Data and HPC solutions on AWS or Contact Us to get started with a workload workshop.
-Shawn Bliesner, Cloud Architect
Business intelligence (BI) is an umbrella term that refers to a variety of software applications used to analyze an organization’s raw data. BI as a discipline is made up of several related activities including data mining, online analytical processing, querying and reporting. Analytics is the discovery and communication of meaningful patterns in data. This blog will look at a few areas of BI that will include data mining and reporting, as well as talk about using analytics to find the answers you need to make better business decisions.
Data Mining is an analytic process designed to explore data. Companies of all sizes continuously collect data, often times in very large amounts, in order to solve complex business problems. Data collection can range in purpose from finding out the types of soda your customers like to drink to tracking genome patterns. To process these large amounts of data quickly takes a lot of processing power, and therefore, a system such as Amazon Elastic MapReduce (EMR) is often needed to accomplish this. AWS EMR can handle most use cases from log analysis to bioinformatics, which are key when collecting data, but AWS EMR can only report on data that is collected, so make sure the collected data is accurate and complete.
Reporting accurate and complete data is essential for good BI. Tools like Splunk’s Hunk and Hive work very well with AWS EMR for modeling, reporting, and analyzing data. Hive is business intelligence software used for reporting meaningful patterns in the data, while Hunk helps interactively review logs with real-time alerts. Using the correct tools is the difference between data no one can use and data that provides meaningful BI.
Why do we collect all this data? To find answers of course! Finding answers in your data, from marketing data to application debugging, is why we collect the data in the first place. AWS EMR is great for processing all that data with the right tools reporting on that data. But more than knowing just what happened, we need to find out how it happened. Interactive queries on the data are required to drill down and find the root causes or customer trends. Tools like Impala and Tableau work great with AWS EMR for these needs.
Business Intelligence and Analytics boils down to collecting accurate and complete data. That includes having a system that can process that data, having the ability to report on that data in a meaningful way, and using that data to find answers. By provisioning the storage, computation and database services you need to collect big data into the cloud, we can help you manage big data, BI and analytics while reducing costs, increasing speed of innovation, and providing high availability and durability so you can focus on making sense of your data and using it to make better business decisions. Learn more about our BI and Analytics Solutions here.
-Brent Anderson, Senior Cloud Engineer
Batch computing isn’t necessarily the most difficult thing to design a solution around, but there are a lot of moving parts to manage, and building in elasticity to handle fluctuations in demand certainly cranks up the complexity. It might not be particularly exciting, but it is one of those things that almost every business has to deal with in some form or another.
The on-demand and ephemeral nature of the Cloud makes batch computing a pretty logical use of the technology, but how do you best architect a solution that will take care of this? Thankfully, AWS has a number of services geared towards just that. Amazon SQS (Simple Queue Services) and SWF (Simple Workflow Service) are both very good tools to assist in managing batch processing jobs in the Cloud. Elastic Transcoder is another tool that is geared specifically around transcoding media files. If your workload is geared more towards analytics and processing petabyte scale big data, then tools like EMR (Elastic Map Reduce) and Kinesis could be right up your alley (we’ll cover that in another blog). In addition to not having to manage any of the infrastructure these services ride on, you also benefit from the streamlined integration with other AWS services like IAM for access control, S3, SNS, DynamoDB, etc.
For this article, we’re going to take a closer look at using SQS and SWF to handle typical batch computing demands.
Simple Queue Services (SQS), as the name suggests, is relatively simple. It provides a queuing system that allows you to reliably populate and consume queues of data. Queued items in SQS are called messages and are either a string, number, or binary value. Messages are variable in size but can be no larger than 256KB (at the time of this writing). If you need to queue data/messages larger than 256KB in size the best practice is to store the data elsewhere (e.g. S3, DynamoDB, Redis, MySQL) and use the message data field as a linker to the actual data. Messages are stored redundantly by the SQS service, providing fault tolerance and guaranteed delivery. SQS doesn’t guarantee delivery order or that a message will be delivered only once, which seems like something that could be problematic except that it provides something called Visibility Timeout that ensures once a message has been retrieved it will not be resent for a given period of time. You (well, your application really) have to tell SQS when you have consumed a message and issue a delete on that message. The important thing is to make sure you are doing this within the Visibility Timeout, otherwise you may end up processing single messages multiple times. The reasoning behind not just deleting a message once it has been read from the queue is that SQS has no visibility into your application and whether the message was actually processed completely, or even just successfully read for that matter.
Where SQS is designed to be data-centric and remove the burden of managing a queuing application and infrastructure, Simple Workflow Service (SWF) takes it a step further and allows you to better manage the entire workflow around the data. While SWF implies simplicity in its name, it is a bit more complex than SQS (though that added complexity buys you a lot). With SQS you are responsible for managing the state of your workflow and processing of the messages in the queue, but with SWF, the workflow state and much of its management is abstracted away from the infrastructure and application you have to manage. The initiators, workers, and deciders have to interface with the SWF API to trigger state changes, but the state and logical flow are all stored and managed on the backend by SWF. SWF is quite flexible too in that you can use it to work with AWS infrastructure, other public and private cloud providers, or even traditional on-premise infrastructure. SWF supports both sequential and parallel processing of workflow tasks.
Note: if you are familiar with or are already using JMS, you may be interested to know SQS provides a JMS interface through its java messaging library.
One major thing SWF buys you over using SQS is that the execution state of the entire workflow is stored by SWF extracted from the initiators, workers, and deciders. So not only do you not have to concern yourself with maintaining the workflow execution state, it is completely abstracted away from your infrastructure. This makes the SWF architecture highly scalable in nature and inherently very fault-tolerant.
There are a number of good SWF examples and use-cases available on the web. The SWF Developer Guide uses a classic e-commerce customer order workflow (i.e. place order, process payment, ship order, record completed order). The SWF console also has a built in demo workflow that processes an image and converts it to either grayscale or sepia (requires AWS account login). Either of these are good examples to walk through to gain a better understanding of how SWF is designed to work.
Contact 2nd Watch today to get started with your batch computing workloads in the cloud.
-Ryan Kennedy, Sr. Cloud Architect
If you’re an old hand in IT (that is, someone with at least 5 years under your belt), making a switch to the cloud could be the best decision you’ll ever make. The question is, do you have what it takes?
Job openings in the cloud computing industry are everywhere – Amazon alone lists 17 different categories of positions related to AWS on its website, and by one estimate there are nearly four million cloud computing jobs just in the United States. But cloud experts, especially architects, are a rare breed. Cloud is a relatively new platform, which limits the number of qualified personnel. Finding people with the right skillsets can be hard.
Cloud architects are experts in designing, running and managing applications in the cloud, leading DevOps teams and working with multiple cloud vendors. A senior cloud architect oversees the broad strategy of a company’s investment in the cloud, and is responsible for managing and delivering ROI from cloud investments and continually aligning with business objectives.
Yet being a cloud architect is not simply about understanding new technologies and features as they come off the conveyor belt. Beyond dealing with rapid technological change, you’ve got to have some creativity and business acumen. If you are fiercely independent and don’t enjoy a little schmoozing with business colleagues and chatting up vendors, this probably is not a good career choice for you. If you don’t like things changing frequently and problem-solving, you may suffer from recurring anxiety attacks.
In talking to customers, we’ve come up with a list of the top non-techie skills that every cloud architect should have. Here are the top 10:
- Strategic problem-solving skills
- Security & compliance experience
- Ability to balance trade-offs with agility
- Business and accounting knowledge
- Customer experience focus
- Deploy & destroy mentality
- Adept negotiation and communications skills
- Ability to solve problems with an eye for the future
- Understanding of platform integrations
- Ability to evolve with the business
In short, cloud architects are like great companions: once you have one, hold on and never let him or her go. Check out the infographic for a complete mapping of the perfect cloud architect!
-Jeff Aden, EVP Business Development & Marketing
With the New Year comes the resolutions. When the clock struck midnight on January 1st, 2015 many people turned the page on 2014 and made a promise to do an act of self-improvement. Often times it’s eating healthier or going to the gym more regularly. With the New Year, I thought I could put a spin on a typical New Year’s Resolution and make it about AWS.
How could you improve on your AWS environment? Without getting too overzealous, let’s focus on the fundamental AWS network infrastructure, specifically an AWS Virtual Private Cloud (VPC). An AWS VPC is a logically isolated, user controlled, piece of the AWS Cloud where you can launch and use other AWS resources. You can think of it as your own slice of AWS network infrastructure that you can fully customize and tailor to your needs. So let’s talk about VPCs and how you can improve on yours.
- Make sure you’re using VPCs! The simple act of implementing a VPC can put you way ahead of the game. VPCs provide a ton of customization options from defining your own VPC size via IP addressing; to controlling subnets, route tables, and gateways for controlling network flow between your resources; to even defining fine-grained security using security groups and network ACLs. With VPCs you can control things that simply can’t be done when using EC2-Classic.
- Are you using multiple Availability Zones (AZs)? An AZ is a distinct isolated location engineered to be inaccessible from failures of other AZs. Make sure you take advantage of using multiple AZs with your VPC. Often time instances are just launched into a VPC with no rhyme or reason. It is great practice to use the low-latency nature and engineered isolation of AZs to facilitate high availability or disaster recovery scenarios.
- Are you using VPC security groups? “Of course I am.” Are you using network ACLs? “I know they are available, but I don’t use them.” Are you using AWS Identity and Access Management (IAM) to secure access to your VPCs? “Huh, what’s an IAM?!” Don’t fret, most environments don’t take advantage of all the tools available for securing a VPC, however now is the time reevaluate your VPC and see if you can or even should use these security options. Security groups are ingress and egress firewall rules you place on individual AWS resources in your VPC and one of the fundamental building blocks of an environment. Now may be a good time to audit the security groups to make sure you’re using the principle of least privilege, or not allowing any access or rules that are not absolutely needed. Network ACLs work at the subnet level and may be useful in some cases. In larger environments IAM may be a good idea if you want more control of how resources interact with your VPC. In any case there is never a bad time to reevaluate security of your environment, particularly your VPC.
- Clean up your VPC! One of the most common issues in AWS environments are resources that are not being used. Now may be a good time to audit your VPC and take note of what instances you have out there and make sure you don’t have resources racking up unnecessary charges. It’s a good idea to account for all instances, leftover EBS volumes, and even clean up old AMIs that may be sitting in your account. There are also things like extra EIPs, security groups, and subnets that can be cleaned up. One great tool to use would be AWS Trusted Advisor. Per the AWS service page, “Trusted Advisor inspects your AWS environment and finds opportunities to save money, improve system performance and reliability, or help close security gaps.”
- Bring your VPC home. AWS, being a public cloud provider, allows you to create VPCs that are isolated from everything, including your on-premise LAN or datacenter. Because of this isolation all network activity between the user and their VPC happens over the internet. One of the great things about VPCs are the many types of connectivity options they provide. Now is the time to reevelautate how you use VPCs in conjunction with your local LAN environment. Maybe it is time to setup a VPN and turn your environment into a hybrid cloud and physical environment allowing all communication to pass over a private network. You can even take it one step further by incorporating AWS Direct Connect, a service that allows you to establish private connectivity between AWS and your datacenter, office, or colocation environment. This can help reduce your network costs, increase bandwidth throughput, and provide a more consistent overall network experience.
These are just a few things you can do when reevaluating your AWS VPC for the New Year. By following these guidelines you can gain efficiencies in your environment you didn’t have before and can rest assured your environment is in the best shape possible for all your new AWS goals of 2015.
-Derek Baltazar, Senior Cloud Engineer
With every new year comes a new beginning. The holidays give us a chance to reflect on our achievements from the previous year, as well as give us a benchmark for what we want to accomplish in the following year. For most individuals, weight loss, quitting a bad habit, or even saving money top the list for resolutions. For companies, the goals are a little bit more straight forward and quantitative. Revenue goals are set, quotas are established, and business objectives are defined. The success of a company is entrenched in these goals and will determine; positively or negatively, how a company should be valued.
Today’s businesses are becoming even more complex than ever, and we can thank new technologies, emerging markets, and the ease of globalization for helping drive these new trends. One of the most impactful and fast adopting technologies that is helping businesses in 2015 is the public cloud.
What’s amazing, though, is that how businesses are planning for the adoption of public cloud is still unknown to most. Common questions such as “Is my team staffed accordingly to handle this technology change?” or “How do I know if I’m architecting correctly for my business?” are coming up often. These questions are extremely common with new technologies, but it doesn’t have to be difficult if you take these simple steps.
- Plan Ahead: Guide your leadership to see that now is the time to review the current technology inventory being utilized by the company and strategically outline what it will take to help the company become more agile, cost effective, and leverage the most robust technologies in the New Year.
- Over communicate: By talking with all the necessary parties, you will turn an unknown topic into water cooler conversation. Involve as many people as possible and ask for feedback constantly. This way, if there is anyone that is not on-board with this technology shift, you will have advocates across the organization helping your cause.
- Track your progress: Keep an active log of your adoption process, pitfalls, to-dos, and active contributors. Establish a weekly cadence to review past success and upcoming agendas. Focus on small wins, and after a while you will see amazing results for your achievements.
- Handle problems with positivity: Technology changes are never easy for an organization, but take each problem as an opportunity to learn. If something isn’t working, it’s probably for good reason. Review what went wrong, learn from the mistakes, and make sure they don’t repeat themselves. Each problem should be welcomed, addressed and reviewed accordingly.
- Stay diligent: Rome wasn’t built in a day, and neither will your new public-cloud datacenter be. Review your plan, do constant check points against your cloud strategy, follow your roadmap and address problems as soon as they come up. By staying focused and tenacious you will be successful in your endeavors.
Happy 2015, and let’s make it a great year.
-Blake Diers, Alliance Manager