Prometheus query quantile of pod memory usage performance

Question

I'd like to get the 0.95 percentile memory usage of my pods from the last x time. However this query start to take too long if I use a 'big' (7 / 10d) range.

The query that i'm using right now is:

quantile_over_time(0.95, container_memory_usage_bytes[10d])

Takes around 100s to complete

I removed extra namespace filters for brevity

What steps could I take to make this query more performant ? (except making the machine bigger)

I thought about calculating the 0.95 percentile every x time (let's say 30min) and label it p95_memory_usage and in the query use p95_memory_usage instead of container_memory_usage_bytes, so that i can reduce the amount of points the query has to go through.

However, would this not distort the values ?

Answer 1

As you already observed, aggregating quantiles (over time or otherwise) doesn't really work.

You could try to build a histogram of memory usage over time using recording rules, looking like a "real" Prometheus histogram (consisting of _bucket, _count and _sum metrics) although doing it may be tedious. Something like:

- record: container_memory_usage_bytes_bucket
  labels:
    le: 100000.0
  expr: |
    container_memory_usage_bytes > bool 100000.0
      +
    (
      container_memory_usage_bytes_bucket{le="100000.0"}
        or ignoring(le)
      container_memory_usage_bytes * 0
    )

Repeat for all bucket sizes you're interested in, add _count and _sum metrics.

Histograms can be aggregated (over time or otherwise) without problems, so you can use a second set of recording rules that computes an increase of the histogram metrics, at much lower resolution (e.g. hourly or daily increase, at hourly or daily resolution). And finally, you can use histogram_quantile over your low resolution histogram (which has a lot fewer samples than the original time series) to compute your quantile.

It's a lot of work, though, and there will be a couple of downsides: you'll only get hourly/daily updates to your quantile and the accuracy may be lower, depending on how many histogram buckets you define.

Else (and this only came to me after writing all of the above) you could define a recording rule that runs at lower resolution (e.g. once an hour) and records the current value of container_memory_usage_bytes metrics. Then you could continue to use quantile_over_time over this lower resolution metric. You'll obviously lose precision (as you're throwing away a lot of samples) and your quantile will only update once an hour, but it's much simpler. And you only need to wait for 10 days to see if the result is close enough. (o:

Answer 2

The quantile_over_time(0.95, container_memory_usage_bytes[10d]) query can be slow because it needs to take into account all the raw samples for all the container_memory_usage_bytes time series on the last 10 days. The number of samples to process can be quite big. It can be estimated with the following query:

sum(count_over_time(container_memory_usage_bytes[10d]))

Note that if the quantile_over_time(...) query is used for building a graph in Grafana (aka range query instead of instant query), then the number of raw samples returned from the sum(count_over_time(...)) must be multiplied by the number of points on Grafana graph, since Prometheus executes the quantile_over_time(...) individually per each point on the displayed graph. Usually Grafana requests around 1000 points for building smooth graph. So the number returned from sum(count_over_time(...)) must be multiplied by 1000 in order to estimate the number of raw samples Prometheus needs to process for building the quantile_over_time(...) graph. See more details in this article.

There are the following solutions for reducing query duration:

• To add more specific label filters in order to reduce the number of selected time series and, consequently, the number of raw samples to process.
• To reduce the lookbehind window in square brackets. For example, changing [10d] to [1d] reduces the number of raw samples to process by 10x.
• To use recording rules for calculating coarser-grained results.
• To try using other Prometheus-compatible systems, which may process heavy queries at faster speed. Try, for example, VictoriaMetrics.