'What is the maximum scrape_interval in Prometheus
I used Prometheus to measure business metrics like:
# HELP items_waiting_total Total number of items in a queue
# TYPE items_waiting_total gauge
items_waiting_total 149
I would like to keep this data for very long term (5 years retention) and I don't need high frequency in scrape_interval. So I set up scrape_interval: "900s".
When I check the graph in Prometheus with 60s resolution, it shows that flapping, but it is not true.
The question is, what is the maximum (recommended) scrape_interval in Prometheus?
Solution 1:[1]
It's not advisable to go above about 2 minutes. This is as staleness is 5 minutes by default (which is what's causing the gaps), and you want to allow for a failed scrape.
Solution 2:[2]
If you want to ignore gaps, it is possible to use some aggregation_over_time functions to get your DATA from Prometheus.
max_over_time(items_waiting_total[900s])
This is useful for situations where frequent gathering of DATA is expensive for collector.
Solution 3:[3]
By default Prometheus fills gaps for up to 5 minutes after each raw sample stored in the database. See these docs for details. If you need to fill bigger gaps between raw samples, then the last_over_time function can help. Just specify the maximum gap duration in square brackets in order to fill gaps. For example, the following query would fill gaps for items_waiting_total time series for up to 900 seconds:
last_over_time(itmes_waiting_total[900s])
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | brian-brazil |
| Solution 2 | Sasha Golikov |
| Solution 3 | valyala |