When it comes to managing and monitoring IT spending, the cloud has created a new layer of complexity. Consider the fact that AWS provides as many as 28,000 service offerings, generating up to millions of billing line items each month. This creates budgeting and planning problems for CIOs because there’s no easy way to interpret what percentage of cloud spending is going toward storage, compute, or network, along with specific applications, projects and services. IT departments also need a way to merge cloud costs with on-premise IT costs to see the full picture of infrastructure spending across key categories. As well, with many individuals from different departments procuring their own AWS resources, a company can have dozens of unmanaged and unlinked accounts. This creates gaps in financial tracking and spend management and prevents a company from taking advantage of volume discounts.
IT needs a unified model to categorize cloud and non-cloud costs together, and automation to map line items into the IT cost model each month. To automate the mapping process, 2nd Watch and Apptio have worked together on a mapping table that specifies where each Amazon product fits within a standard cost model. This mapping is now embedded in the Apptio Cost Transparency application, a solution for integrating AWS usage with billing, cost categorization, modeling of total costs including internal labor and self-service analytics. This allows IT organizations to categorize cloud costs into trackable categories such as Cloud Windows in Compute or Cloud Archive in Storage.
Determining the run costs of an application is another goal, and requires mapping cloud resources, such as servers and storage, to individual applications. Many IT organizations have not yet adapted their management processes to track application relationship data for cloud infrastructure. Linked accounts and tagging are two ways to get around these hurdles on AWS.
Many enterprises have several AWS accounts at the team and departmental levels in order to encourage agility, but these unlinked accounts create gaps in cost and operational management. To unify unlinked accounts across an organization, companies can use the Apptio application to link individual accounts into one “master account” paid through an IT cost center. This provides visibility into enterprise spend on AWS yet still maintains business-unit level tracking. It also enables savings with volume discounts, that’s not possible when spending goes across several individual AWS accounts.
AWS tags help group usage and expenses across shared key resources like databases. Tagging helps accomplish the problem of mapping AWS resources back to specific business projects, such as “Marketing Web Staging” and “Marketing Web Production.” Detailed tagging can help answer questions such as, how much of an entire application portfolio is comprised of AWS services or what percentage of which projects are using cloud resources? One thing to keep in mind is that AWS tags are applied only to individual accounts. AWS tagging is ideal for environments where you need to share resources across multiple workloads.
There are some limitations to this manual approach for managing individual accounts and tagging, however – managing numerous logins and passwords, going through the AWS setup process for each individual account, creating and controlling a tagging schema, etc. For a more scalable approach to managing AWS accounts and tagging, consider solutions like our 2W Insight billing application, which enables grouping of tags across AWS accounts and provides tools to track and analyze cloud costs by cost center, business unit department, etc. For more information on 2W Insight, contact us.
To learn more about best practices for managing and tracking cloud spending, download our Analyzing Cloud Costs white paper.
-Jeff Aden, EVP, Marketing & Strategic Business Development
AWS enables enterprises to trade capital expense for variable expense, lower operating costs and increase speed and agility. As enterprises begin to deploy cloud services across their business, it is critical to have a standardized approach to allocate usage costs to the appropriate department or cost center. By tracking costs at the cost center level, Enterprises gain visibility throughout their organization – and specifically who is spending precious IT funds.
To allocate costs, usage must first be grouped. AWS provides two methods to group usage; Resources Tags and AWS accounts. Each method is useful but also comes with downsides.
Using AWS Tagging to group usage
- Grouping by tag enables enterprises to run all of their workloads (applications) in a single AWS account, simplifying management within the AWS console.
- A tagging schema needs to be created, universally deployed and tightly controlled.
- Care has to be taken to ensure all individual AWS resources are tagged properly as any mistake in tagging will cause a resource to be left out of the group and not reported properly.
- Many AWS resources are un-tagable, which will require the creation and maintenance of a separate cost distribution scheme to allocate those costs across the enterprise.
- Reserved Instance (RI) discounted usage pricing cannot be linked to a single tag group and can result in significant costing inaccuracies.
Using Multiple AWS Accounts to group usage
- Using individual AWS accounts for each workload provides the most accurate and detailed reporting of costs and usage.
- By creating a separate AWS account for each workload, enterprises can track all associated costs (including RIs) and allocate them to cost centers, departments and/or business units.
- When using AWS accounts to group usage, each account must be manually set up.
- There is no method of sharing resources, such as databases, with multiple workloads as each workload is located in separate AWS accounts.
Given the challenges of both “account based” and “tag based” grouping, we have found that the tracking methodology needs be aligned to the applications or workloads. For deployments where the resources are 100% dedicated to a specific workload, grouping by AWS accounts is ideal as it is the only way to ensure fully accurate costing. Using AWS tagging should be used when you need to share resources across multiple workloads, however enterprises must note that the costing will not be 100% accurate when using tag groups.
Tracking and Allocating Costs for Workloads with Dedicated Resources
As stated above, workloads that do not need to share resources should be set up in unique AWS accounts. This can be accomplished by establishing individual AWS accounts for each workload and mapping them directly to your enterprise organizational structure. The example below illustrates how a complex enterprise can organize its cloud expenses and provide showback or chargeback reports across the enterprise.
In this example, the enterprise would receive two bills for their cloud usage – Business Unit 1 and Business Unit 2. Under each business unit there are multiple levels of cost centers that roll up to each subsequent higher level – which is typical with many Enterprise organizations. In this example, AWS accounts are created for each project/workload then rolled up to provide consolidated usage information by department and business unit. This type of structure enables:
- The owners at the “resources and workload cost accrual and tracking” levels to track their individual usage by AWS accounts, which captures 100% of the cost associated with each AWS account
- The management of department level to view the consolidated usage for their respective cost centers and workloads
- The management of each business unit to view usage by department and AWS account and receive a bill for its entire consolidated usage
This provides a reliable and accurate methodology to track and allocate cloud usage based on your distinct enterprise organizational structure. It does, however, require a disciplined approach to creating new projects and updating your expense management platform to provide executive-level dashboards and the ability to drill-down to detailed consumption reports by cost center. This enables Enterprise IT to provide executive-level transparency while keeping excessive resource consumption under control and reduce IT costs.
Tracking and Allocating Costs for Workloads with Shared Resources
In many organizations there is a need to share key resources, such as databases, across multiple workloads. In these cases it is a best practice to use AWS tags to group your expenses. This method requires careful set up of resources and the creation of a schema to allocate shared resources and resources that cannot be tagged across the enterprise.
Tagging allows enterprises to assign its own metadata to each tag-able resource. Tags do not have any semantic meaning to AWS resources and are interpreted strictly as a string of characters. Tags are made up of both a “Key” and a “Value”. AWS allows up to 10 Keys for each resource, and each Key can have can have unlimited values enabling very detailed grouping of resources. Tagging should be set up based on the needs of the organization and the AWS architecture design. The image below illustrates how to establish a tagging scheme for a 2-Tier Auto-scalable Web Application.
As the project moves from Web Sandbox to Web Staging to Web Production, you can use tags to track usage. When the application is in the Sandbox all resources are tagged with the key “Web Sandbox” and the appropriate value (Environment, Owner, App and/or IT Tower). When the project moves to “Web Staging” you simply replace the original key and values with the ones associated with the next step in development.
While there is no one-size-fits-all solution to AWS expense management, deploying one or both of these methods can provide you the visibility necessary to successfully meet the tracking and analytical needs of your enterprise.
-Tim Hill, Product Manager