1-888-317-7920 info@2ndwatch.com

AWS S3-Glacier Lifecycle Management

Not long ago, 2nd Watch published an article on Amazon Glacier. In it Caleb provides a great primer on the capabilities of Glacier and the cost benefits.  Now that he’s taken the time to explain what it is, let’s talk about possible use cases for Glacier and how to avoid some of the pitfalls.  As Amazon says, “Amazon Glacier is optimized for data that is infrequently accessed and for which retrieval times of several hours are suitable.”  What immediately comes to mind are backups, but most AWS customers do this through EBS snapshots, which can restore in minutes, while a Glacier recall can take hours.  Rather than looking at the obvious, consider these use cases for Glacier Archival storage: compliance (regulatory or internal process), conversion of paper archives, and application retirement.

Compliance often forces organizations to retain records and backups for years, customers often mention a seven year retention policy based on regulatory compliance.  In seven years, a traditional (on premise) server can be replaced at least once, operating systems are upgraded several times, applications have been upgraded or modified, and backup hardware/software has been changed.  Add to that all the media that would need to be replaced/upgraded and you have every IT department’s nightmare – needing to either maintain old tape hardware or convert all the old backup tapes to the new hardware format (and hope too many haven’t degraded over the years).  Glacier removes the need to worry about the hardware, the media, and the storage fees (currently 1¢ per GB/month in US-East) are tiny compared to the cost of media and storage on premise.  Upload your backup file(s) to S3, setup a lifecycle policy, and you have greatly simplified your archival process while keeping regulatory compliance.

So how do customers create these lifecycle policies so their data automatically moves to Glacier?  From the AWS Management Console, once you have an S3 bucket there is a Property called ‘Lifecycle’ that can manage the migration to Glacier (and possible deletion as well).  Add a rule (or rules) to the S3 bucket that can migrate files based on a filename prefix, how long since their creation date, or how long from an effective date (perhaps 1 day from the current date for things you want to move directly to Glacier).  For the example above, perhaps customers take backup files, move them to S3, then have them move to Glacier after 30 days and delete after 7 years.

Lifecycle Rule

Before we go too far and setup lifecycles, however, one major point should be highlighted: Amazon charges customers based on GB/month stored in Glacier and a one-time fee for each file moved from S3 to Glacier.  Moving a terabyte of data from S3 to Glacier could cost little more than $10/month in storage fees, however, if that data is made up of 1k log files, the one-time fee for that migration can be more than $50,000!  While this is an extreme example, consider data management before archiving.  If at all possible, compress the files into a single file (zip/tar/rar), upload those compressed files to S3 and then archive to Glacier.

-Keith Homewood, Cloud Architect


Elasticity: Matching Capacity to Demand

According to IDC, a typical server utilizes an average of 15% of its capacity. That means 85% of a company’s capital investment can be categorized as waste. While virtualization can increase server capacity to as high as 80%, the company is still faced with 20% waste under the best case scenario. The situation gets worse when companies have to forecast demand for specific periods; e.g., the holiday season in December. If they buy too much capacity, they overspend and create waste. If they buy too little, they create customer experience and satisfaction issues.

The elasticity of Amazon Web Services (AWS) removes the need to forecast demand and buy capacity up-front. Companies can scale their infrastructure up and down as needed to match capacity to demand. Common use cases include: a) fast growth (new projects, startups); b) on and off (occasionally need capacity); c) predictable peaks (specific demand at specific times); and d) unpredictable peaks (demand exceeds capacity). Use the elasticity of AWS to eliminate waste and reduce costs over traditional IT, while providing better experiences and performance to users.

-Josh Lowry, General Manager Western U.S.



Believe the Hype

A lot of companies have been dipping their toe in the proverbial Cloud waters for some time, looking at ways to help their businesses be more efficient, agile and innovative. There have been a lot of articles published recently about cloud being overhyped, cloud being the new buzzword for everything IT, or about cloud being just a fad. The bottom line is that cloud is enabling a completely new way to conduct business, one that’s not constrained but driven through a completely new business paradigm and should not be overlooked but leveraged, and leveraged immediately.

    • Cyclical Business Demand – We’ve been helping customers architect, build, deploy and manage environments for unpredictable or spikey demand. This has become more prevalent with the proliferation of mobile devices and social media outlets where you never know when the next surge in traffic will come.
    • Datacenter Transformation – Helping customers figure out what can move to the public cloud and what should stay on premise is a typical engagement for us. As the continued migration from on premise technology to cloud computing accelerates, these approaches and best practices are helping customers not just optimize what they have today but also ease the burden of trying to make an all or nothing decision.
    • Financial Optimization – Designing a way to help customers understand their cloud finances and then giving them the ability to create financial models for internal chargebacks and billing can sometimes be overlooked upfront. We’ve developed solutions to help customers do both where customers are seeing significant cost savings.



Amazon Glacier: Your 1¢ Cloud-based Backup Solution

If you haven’t heard of Amazon Glacier, you need to check it out. As its name implies, you can think of Glacier as “frozen” storage. When considering the speed of EBS and S3, Glacier by comparison moves glacially slow. Consider Glacier as essentially a cloud-based archival solution that works similarly to old-style tape backup. In the past, backups first ran to tape, then were stored locally in case of immediate access requirements, and were then taken off-site once a certain date requirement was met (once a week, once a month, etc.). Glacier essentially works as the last stage of that process.

When a snapshot in S3, for instance, gets to be a month old, you can instruct AWS to automatically move that object to Glacier. Writing it to Glacier happens pretty much immediately, though being able to see that object on your Glacier management console can take between 3-5 hours. If you need it back, you’ll issue a request, but that can take up to 24 hours to be resolved. Amazon hasn’t released the exact mechanics of how they’re storing the data on their end, but large tape libraries are a good bet since they jive with one of Glacier’s best features: its price. That’s only $0.01 per gigabyte. Its second best feature is 11 nines worth of “durability” (which refers to data loss) and 4 nines worth of “reliability” (which refers to data availability). That’s 99.999999999% for those who like the visual.

Configuring Glacier, while a straightforward process, will require some technical savvy on your part. Amazon has done a nice job of representing how Glacier works in an illustration:


As you can see, the first step is to download the Glacier software development kit (SDK), which is available for Java or .NET.  Once you’ve got that, you’ll need to create your vault. This is an easy step that starts with accessing your Glacier management console, selecting your service region (Glacier is automatically redundant across availability zones in your region, which is part of the reason for its high durability rating), naming your vault, and hitting the create button. I’m using the sandbox environment that comes with your AWS account to take these screen shots, so the region is pre-selected. In a live environment, this would be a drop-down menu providing you with region options.


The vault is where you’ll store your objects, which equate to a single file, like a document or a photo. But instead of proceeding directly to vault creation from the screen above, be sure and set up your vault’s Amazon Simple Notification Service (SNS) parameters.


Notifications can be created for a variety of operations and delivered to systems managers or applications using whatever protocol you need (HTML for a homegrown web control or email for your sys admin, for example). Once you create the vault from the notifications screen, you’re in your basic Glacier management console:


Uploading and downloading documents is where it gets technical. Currently, the web-based console above doesn’t have tools for managing archive operations like you’d find with S3. Uploading, downloading, deleting or any other operation will require programming in whichever language for which you’ve downloaded the SDK. You can use the AWS Identity and Access Management (IAM) service to attach user permissions to vaults and manage billing through your Account interface, but everything else happens at the code level. However, there are third-party Glacier consoles out there that can handle much of the development stuff in the background while presenting you with a much simpler management interface, such as CloudBerry Explorer 3.6. We’re not going to run through code samples here, but Amazon has plenty of resources for this off its Sample Code & Libraries site.

On the upside, while programming for Glacier operations is difficult for non-programmers, if you’ve got the skills, it provides a lot of flexibility in designing your own archive and backup processes. You can assign vaults to any of the various backup operations being run by your business and define your own archive schedules. Essentially, that means you can configure a hierarchical storage management (HSM) architecture that natively incorporates AWS.

For example, imagine a typical server farm running in EC2. At the first tier, it’s using EBS for immediate, current data transactions, similar to a hard disk or SAN LUN. When files in your EBS store have been unused for a period of time or if you’ve scheduled them to move at a recurring time (like with server snapshots), those files can be automatically moved to S3. Access between your EC2 servers and S3 isn’t quite as fast as EBS, but it’s still a nearline return on data requests. Once those files have lived on S3 for a time, you can give them a time to live (TTL) parameter after which they are automatically archived on Glacier. It’ll take some programming work, but unlike with standard on-premises archival solutions, which are usually based on a proprietary architecture, using Java or .NET means you can configure your storage management any way you like – for different geographic locations, different departments, different applications, or even different kinds of data.

And this kind of HSM design doesn’t have to be entirely cloud-based. Glacier works just as well with on-premises data, applications, or server management. There is no minimum or maximum amount of data you can archive with Glacier, though individual archives can’t be less than 1 byte or larger than 40 terabytes. To help you observe regulatory compliance issues, Glacier uses secure protocols for data transfer and encrypts all data on the server side using key management and 256-bit encryption.

Pricing is extremely low and simple to calculate. Data stored in Glacier is $0.01 per gigabyte. Upload and retrieval operations run only $0.05 per 1000 requests, and there is a pro-rated charge of $0.03 per gigabyte if you delete objects prior to 90 days of storage. Like everything else in AWS, Glacier is a powerful solution that provides highly customizable functionality for which you only pay for what you use. This service is definitely worth a very close look.

-Caleb Carter, Solutions Architect


How to Spin Up an AWS Virtual Machine

If you are a user like me, it may actually take you longer to make a cup of coffee than creating an AWS account. Go to the AWS EC2 login site to create an account. You’ve got 12 months of free tier service with your new account, and you can cancel if you decide it’s not for you. Careful, however, if you ignore your VM for more than three months, Amazon may decide the issue for you, though they’ll send you an email warning 30 days prior. Account created? Let’s build a server!

SpinUpServer 1

Hit the Quick Launch Wizard (middle left) and you’ve got a few things to fill out before your VM comes to life. First, name your server (top), then name your secure key (middle) and finally choose the operating system you’d like to use. For now, just choose the Amazon Linux AMI since some of the other choices, notably Windows Server, have an added cost even during your free tier. Make sure to download your security key and save it where you can find it again.

SpinUpServer 2

Once you’ve selected your basic options, you’ll get your first glimpse of the EC2 dashboard (above). See the big blue “Launch Instance” button? Hit that. You’ll see a screen informing you that your sever is being created. It’s time to get that cup of coffee. When you come back, you can click back to your EC2 dashboard view and see:

EC2 Dashboard3

That’s it, you’re all set! You can start using  your new server immediately. There are a couple different SSH options available, depending on whether you’re using Linux or Windows. If you use SSH through Linux, all you need is to go to your terminal line and type sudo apt-get install openssh-server. But if you’re a Windows fan, there are plenty of free add-on SSH packages, such as Cygwin or PuTTY. To get accesst to your server via SSH, there are some pieces of information you’ll need, but Amazon has made them easy to find. Just check the box next to your server in the screen above, and a scroll window pops up underneath with all the information you’ll need, including your VM’s name, your private IP address, and more:

SpinUpServer 4

If you’re unsure what you’d like to do with your new VM, AWS has excellent and easy-to-digest Getting Started guides that cover all the bases:

AWS Getting Started5

The above example walks you through one of the first steps you should take, which is deciding who has access to your VM. There are also guides on locating and using your IP address, configuring your firewall (don’t worry there’s a default setting), setting up roles, user groups, and management certificates, as well as implementing any of the many services that come with your Free Tier package:

SpinUpServer 6

To remove your new VM, just return to that EC2 management screen and click the Instance Action link. From there hit Terminate and say yes to the “Are you sure?” question. That’s all there is to it.

Try out AWS on some of your in-house application workloads. Check out the default monitoring and alert services. Move data back and forth and record response times. Create a few more VMs and use them to build a basic virtual server farm. Try some of the more popular services shown above like S3, CloudWatch, or Amazon RDS. In other words, take the time to dig into AWS and really see what it can do for you and your organization. As always feel free to ask us at 2nd Watch questions as you go along. We are happy to help!


Flexible Payments

There are many companies selling goods or services on the AWS platform, and all have to solve an important question: How do I get paid? As with most elements of cloud services, AWS has you well covered; if anything maybe a little too well covered considering the rather large array of options and solutions they make available.

That list is rather long, but most fall beneath the umbrella of the AWS Flexible Payments Service (FPS). You’ll find that FPS has a long list of sub-offerings, but at the start you can break it down into two basic options: The FPS web service and Amazon Simple Pay.

The FPS web service boils down to a set of APIs that allow you to integrate the FPS service into your site or web app, which gives you the ability to implement a fairly wide variety of payment options. Customers will experience a co-branded Amazon FPS user interface, which means they’ll jump from your site or application to a site that’s co-branded by you and Amazon when it’s time to pay.

Customers pay via the co-branded UI and you receive an authorization from Amazon, usually called a token ID, which you can use to initiate the payment. Once you’ve got the token ID, you can set up a variety of payments, including a one-time payment, a reserved payment, or a subscription, as well as options for settling payments, canceling transactions or providing refunds. Amazon receives the money from your customers and deposits it into your Amazon Payments account where you can do with it as you like. You can track payments off your Amazon account page, or receive instant payment notifications. All in all, it’s a very robust service.

The Talk Market is an AWS customer that’s a great example of FPS in action. The company has recently brought a hosted shopping channel to the web, where small and large e-tailers can promote products via video. The Talk Market has been using Amazon FPS since 2008 to build and direct a seamless channel between its customers (the vendors) and their customers with a fee taken out for Talk Market. Vendors sign up and The Talk Market automatically provides them with a credit card processor that drops their sales revenue into their Amazon account. Everything is managed by FPS.

Implementing FPS definitely requires the help of a developer. Using the FPS APIs requires a working knowledge of your site or web app code as well as XML and another web-capable programming language (like Java, JavaScript, or Python among others). Developers need to build the routing relationship with FPS as well as a secure key relationship (which controls security between the user, FPS, and your operation). Amazon has provided quite a bit of developer resource material, including lots of Quick Starts and code samples as well as the FPS Sandbox, developer forums and FAQs.  If you’re getting the feeling that this isn’t something you can learn in a weekend, you’re right. Expertise and customization are a necessary requirement of FPS since it has to be customized and requires additional code depending on which services of FPS you’re using.

For people who can’t afford expert programming help, like small startups, there’s Amazon Simple Pay. If you need to sum this up quickly, it’s FPS but with Amazon handling everything for you in the background. All you really need to do is add a “pay” button to your app or site that links the customer back to AWS. After that, it’s in Amazon’s hands. The transaction is handled on its servers and the user is then directed back to your site, but without any kind of token. Amazon has completed the entire transaction and simply drops your money into your Amazon Payment account. Amazon Simple Pay is a great starter service, though many businesses will quickly outgrow it.

But wait! There’s more! One of the best add-on options for an FPS implementation is Checkout by Amazon (CbA). This is a checkout interface option based on the FPS platform that you can add to your site with some code snippets. When customers decide it’s time to pay, they click on “checkout” at which point they’ll go through either the inline or standard CbA experience, depending on what you’ve implemented. Both options ask the customer to log into their Amazon account, but inline lets them access shipping and payment options from inside your own checkout experience. In standard, they’re presented with a separate pop-up window that’s very similar to the Amazon checkout page (including branding). This lets them choose from the payment and shipping options they have stored in their Amazon accounts. CbA also offers a full payments service that handles all the details, like shipping charges, sales tax, refunds, and more with payment reporting. There’s even a mobile version, Checkout for Amazon Mobile that lets people shop off tablets and smartphones.

Sound like a lot? It is. And we still haven’t covered everything. You’ve got many other options available, including Fulfillment by Amazon, (an Amazon-run warehouse and shipping service), TextPayMe (send payments via text message), Amazon Payments for Nonprofits, Simple Pay Subscriptions, and many more. While Amazon has done a fairly good job making most of these services accessible individually, they’re still difficult to combine into a payments solution that’s right for your particular business.

You can start with Amazon Simple Payments, but as your business grows, you’re going to need something that’s more suited to your individual needs. When you get there, it’s time to work on an FPS strategy. That means deciding how you want FPS implemented, which options should be available to your customers, what you’ll need in the way of transaction reporting, how to maintain FPS as your site or application evolves, and any additional features you might need that only third-party partners can provide.

Without careful planning, FPS can become difficult or seem insufficient. Then you’ll be looking for another solution that’s likely to be less robust and cost more, when FPS could have fulfilled your every need and then some if you’d had the proper planning in place. At 2nd Watch we use Flexible Payments on our own 2W SharePoint offering to provide seamless payment options for our users. When it’s time to get serious about a payments solution, we can help you, too.

-Kelly Sale, Senior Developer