How can I query prometheus data greater than value but include all data in series vector?

Question

I think this image says it all

I want to query a series in Prometheus with values greater than .5 but include the lower values in the series so the chart is complete. These are Gauge metrics ranging from 0-1 (decimal percent/ratio) I would like for all the lines in the chart to be complete. Current Query

avg_over_time((failure_percentage > .5)[10m:])

I have tried self joins and grouping to no success.

Answer 1

I was able to solve this with the help of some folks on the prometheus slack using a sort of join hack

avg_over_time(failure_percentage[10m]) * ( (failure_percentage > 0.5) ^0 )

Original Comment by user viq (For full context and explanation)

I wonder.... a drity trick that comes to mind is something like

 metric * ( (metric > 0.5) ^ 0)

Since for multiplication to work both sides need to exactly match on labels, so you'll get only the results that match what's on the right, right (I think) should give you only results that match the condition, and ^0 should make the value always be 1, so you're getting in effect metric * 1 maaaaybe, untested

Answer 2

As I know it is impossible to return all the data points for time series that exceed the given value with PromQL.

But this is possible with MetricsQL:

with (q = failure_percentage) q if ((q > 0.5) default 0)

It is possible to create WITH template function that returns all the points for time series that exceed the given threshold:

with (
  f(q, threshold) = q if ((q > threshold) default 0)
)
f(failure_percentage, 0.5)

Answer 3

It's now possible with the help of the PromQL @ modifier

First for each series we want to get the maximum value over the plot interval. For that we are going to leverage the @ end() modifier that allows to take the value of the last evaluation interval. We have to set the interval to be the same size as the plot to ensure we get the max of all the displayed values.

max_over_time((failure_percentage)[10m:] @ end())
max_over_time((failure_percentage)[$__range:] @ end()) (in Grafana to be more dynamic)

Then we transform those flat series in two groups : the ones that are above the 0.5 threshold to 1 and those below the threshold to 0 using the >bool operator. And keep only the former one with == 1.

max_over_time((failure_percentage)[10m:] @ end()) >bool 0.5 == 1

Finally we use the * operator (as multiplying by one is noop) to join the selected series with the original one.

failure_percentage * (max_over_time((failure_percentage)[10m:] @ end()) >bool 0.5 == 1)

Grafana pannel example