Combine Prometheus alert conditions with "or"?

Question

I know Prometheus supports and and unless in alert statements: https://www.robustperception.io/combining-alert-conditions/

Is there support for or? I want to write an alert that fires if a timeseries has crossed a threshold or is absent for a given period of time.

Answer 1

Yes, or is a Prometheus operator. See the docs.

Answer 2

Prometheus supports or operation as already pointed in this answer. The following query must be used for alerting when a time series foo exceeds the given threshold or if a time series foo is absent for the duration d:

foo > threshold or absent_over_time(foo[d])

See absent_over_time() docs. Note that the absent_over_time() doesn't work as expected, e.g. doesn't return non-empty result if there are multiple time series with foo name and only a single time series has no new samples during the duration d. Basically, it works as expected only if there is only a single matching time series.

Prometheus doesn't provide functionality for creating universal alerts, which could detect time series without new samples over the given duration across multiple matching time series. However, such function exists in VictoriaMetrics - see lag(). For example, the following query alerts (e.g. returns non-empty result) for time series with foo name, which exceeded the given threshold or which didn't have new samples during the last 5 minutes, while they had at least a single sample during the last hour:

foo > threshold or lag(foo[1h]) > 5m