'How to get the 95th percentile of an average in Prometheus?

So I'm aware of some percentile functions in PromQL like histogram_quantile which is used in a case like this:

// Over the past 5 minutes, what's the maximum http response time experienced by 95% of our users
histogram_quantile(0.95, rate(http_request_duration_bucket[5m])

And we can calculate the average this way:

// Over the past 5 mins, what the average http response time?
avg by (webId) (rate(http_request_duration_sum[5m])/rate(http_request_duration_count[5m])

Is it possible to combine these two function to get the query that means the following: Over the past 5 mins, what's the maximum average HTTP response time experienced by 95% of our users? AKA 95 percentile of the AVERAGE?

I tried something like:

histogram_quantile(0.95, avg by (webId) (rate(http_request_duration_sum[5m])/rate(http_request_duration_count[5m]))

But it doesn't seem to work. Any suggestions or gaps in my understanding?

prometheusgrafanapromql


Solution 1:[1]

Try the following query:

quantile(0.95, avg by (webId) (rate(http_request_duration_sum[5m])/rate(http_request_duration_count[5m])))

It uses quantile() aggregate function for calculating the given quantile over average response times calculated per each webId.

Solution 2:[2]

I found the answers above didn't work for my situation, but quantile_over_time allowed me to extract a percentile from a sum:

quantile_over_time(0.95,
  (sum by (component_name) (
     node_memory_MemTotal_bytes 
     - node_memory_MemFree_bytes 
     - node_memory_Cached_bytes)[2w:]
  )
)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 valyala
Solution 2 RobM

Related Questions