Transparency in pricing and consistency in its application are a big part the AWS success story. Each service is priced individually, clearly listed on their web page. The only discounts available are for pre-booking and pre-paying for resources – typically as Reserved Capacity (called Reserved Instances in EC2). In this blog we will go over some basics of AWS billing that you should be aware of as you purchase capacity from AWS.
Billing Basics
Each AWS service is billed separately, using its own metric. Most services are straightforward – e.g. an EC2 instance is billed per second; some are a bit more complex – e.g. Lambda is priced both per invocation and per GB-sec used.
At any point, you can log into the AWS billing console and see your approximate invoice for the month to date. AWS billing data is not real-time – there is a lag between usage and billing. AWS publish no guidance on the matter; our experience has been that AWS updates its billing data at least once a day.
At the end of the month, a final invoice is generated that includes any refunds, credits or support fees. This can be viewed in the management console, printed or downloaded as a CSV file. The below figure is a screenshot of a page (page 1,2 of 16) for aa AWS account with minimal usage. The invoice is rather opaque – it shows aggregate charges incurred by service, but there is no way to map that on to your AWS resources and analyze the spend.
Cost and Usage Report
AWS does allow you to access the raw data that was used to generate the invoice. This is called the Cost and Usage Report (CUR) and can be dumped to an S3 bucket into your account. This capability is not enabled by default and should to be set up in the billing console. Data in this file is critical to getting a better view of AWS Costs; the file is refreshed up to three times a day.
The reports themselves are free, but S3 storage rates do apply. When you create a report, you can choose to have each new report overwrite the previous one or creating a new version of the report. The files are dense, but not large in size.
This setup should be done from your master payer account.
AWS documents the steps to creating this report here. Keep in mind that all CUR files are generated from a specific AWS account (3XXX), so a specific bucket policy must applied to the bucket created for for this report. While AWS does provide a report creation wizard that checks this setup, it won’t give you useful feedback if this policy isn’t set up, so please make sure you’ve done this step.
The CUR format relatively new and is a big step up from the previous file format (the Detailed Billing Report or DBR), which could easily go into the millions of lines and still not have enough information for cost optimization – e.g. Reserved Instance utilization was not available with the previous format and had to be calculated separately.
The file is still pretty dense – see the screenshot below for a sample hourly CUR from one of our test environments (the account numbers have been pixelated). This file has over 44,000 lines of data, with each line having 112 columns of information. If you are ok with less granularity, you can use the daily report; the daily CUR report for the same test environment is still 25,000 lines of data. We just created a free tier account with almost no resources in it, and that CUR turns out to be 3500 lines long.
Clearly, that these files were not designed to be human-readable; there are a few options to make sense of the data:
- Build Your Own – The amount of data generated will easily overwhelm an Excel sheet; you can dump the data into a SQL database for analysis (you can also use Amazon Athena to query S3 directly). There are a number of open source projects that can help you make sense of the data
- Using AWS built-in tools – AWS Cost Explorer (free) supports visualization and AWS Trusted Advisor (charged as percentage of total spend) provides cost optimization recommendations.
- Using a 3rd party Cloud Expense Management tool – HyperCloud provides this capability through HyperCloud Analytics and we will be discussing it in this article and a few follow-on articles as well
Using Third Party Tools Get Access To Billing Data
Both AWS and all third-party tools work off the same data – either the Cost and Usage Report (CUR) or the older Detailed Billing Report (DBR). AWS is retiring the DBR format, and third-party tools are encouraged to move over to using CUR exclusively.
The CUR file has details on every AWS resource (instances, reserved instances, volumes, buckets, bandwidth etc.), their utilization, details on how it was invoked, which AZ it was run in etc. All this data is referenced by its resource ID (EC2 instances) or ARN.
AWS updates this data up to three times a day, meaning the bill is never more than a few hours out of date with actual utilization. At the end of the month, AWS factors in support costs, any refunds or other credits and updates the CUR files if needed.
In follow on articles, we will discuss AWS Billing and Cost Management in more detail and compare the built-in tools with our own cost management tool. We will also share best practices we have develop years by working with AWS – both as users ourselves and tool smiths.
