'CloudWatch Cost - Data Processing
I'd like to know if possible to discover which resource is behind this cost in my Cost Explorer, grouping by usage type I can see it is Data Processing bytes, but I don't know which resource would be consuming this amount of data. Have some any idea how to discover it on CloudWatch?
amazon-web-servicesSolution 1:[1]
This is almost certainly because something is writing more data to CloudWatch than previous months.
As stated this AWS Support page about unexpected CloudWatch logs bill increases:
Sudden increases in CloudWatch Logs bills are often caused by an increase in ingested or storage data in a particular log group. Check data usage using CloudWatch Logs Metrics and review your Amazon Web Services (AWS) bill to identify the log group responsible for bill increases.
Your screenshot identifies the large usage type as APS2-DataProcessing-Bytes. I believe that the APS2 part is telling you it's about the ap-southeast-2 region, so start by looking in that region when following the instructions below.
Here's a brief summary of the steps you need to take to find out which log groups are ingesting the most data:
How to check how much data you're ingesting
The IncomingBytes metric shows you how much data is being ingested in your CloudWatch log groups in near-real time. This metric can help you to determine:
- Which log group is the highest contributor towards your bill
- Whether there's been a spike in the incoming data to your log groups or a gradual increase due to new applications
- How much data was pushed in a particular period
To query a small set of log groups:
- Open the Amazon CloudWatch console.
- In the navigation pane, choose Metrics.
- For each of your log groups, select the IncomingBytes metric, and then choose the Graphed metrics tab.
- For Statistic, choose Sum.
- For Period, choose 30 Days.
- Choose the Graph options tab and choose Number.
- At the top right of the graph, choose custom, and then choose Absolute. Select a start and end date that corresponds with the last 30 days.
For more details, and for instructions on how to query hundreds of log groups, read the full AWS support article linked above.
Solution 2:[2]
Apart from the steps which Gabe mentioned what helped me identify the resource which was creating large number of logs was by:
- heading over to Cloudwatch
- selecting the region which showed in Cost explorer
- Selecting Log Groups
- From settings under Log Groups, Enabling column Stored bytes to be visible
This showed me which service was causing a lot of logs to be written to Cloudwatch.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Gabe Hollombe |
| Solution 2 | Vaulstein |