Compare 2 metrics in Prometheus

Question

I'm new in prometheus, need help.

I have custom metric on my 2 servers (just shows version of application):

app_version{plant="dev",env="demo"} 55.119
app_version{plant="dev",env="live"} 55.211

I want to compare this metrics and send ALERT if they are not equal trying for smth like this:

alert: Compare
expr: app_version{env="demo"} != app_version{env="live"}
for: 5s
labels:
  severity: page
annotations:
  summary: Compare

and this alert is green. What the right way to compare 2 metrics?

Answer 1

The different value for the env label means that there's nothing to match on each side of the expression, so it returns nothing and there's no alerts. You can adjust this behaviour using ignoring:

app_version{env="demo"} != ignoring (env) app_version{env="live"}

Answer 2

Prometheus performs the app_version{env="demo"} != app_version{env="live"} query in the following way:

1. It selects time series matching the app_version{env="demo"}. All these time series have the name app_version and the label env="demo" according to time series selector rules.
2. It selects time series matching the app_version{env="live"}. All these time series have the name app_version and the label env="live".
3. It searches for time series pairs on the left and the right side of != operator with identical sets of labels according to these rules.
4. It compares the found time series pairs with != operator.

Unfortunately there are no time series pairs with identical sets of labels for the query app_version{env="demo"} != app_version{env="live"}, since all the time series on the left side have env="demo" label, while all the time series on the right side have env="live" label. This can be fixed by instructing Prometheus to ignore env label when finding the matching time series according these docs. There are the following options:

1. To enumerate labels, which must be ignored when searching for time series pairs with identical labelsets, via ignoring() modifier:

app_version{env="demo"} != ignoring(env) app_version{env="live"}

It is likely this query will return an empty result too, since app_version time series may differ by other labels such as instance (see these docs about instance label). While it is possible to add the instance label into ignoring() list, the next option may work better.

2. To enumerate labels, which must be taken into account when searching for time series pairs with identical labelsets, via on() modifier. For example, the following query would take into account only job label when finding time series pairs with identical labelsets:

app_version{env="demo"} != on(job) app_version{env="live"}

This query may result in multiple matches for labels or many-to-many matching not allowed errors if multiple time series on one side of != operator have the same set of labels from on() list. For example, if Prometheus scrapes app metrics from multiple targets, then app_version{env="live"} may match multiple time series, which are scraped from different targets:

app_version{env="live",job="my-app",instance="host1"}
app_version{env="live",job="my-app",instance="host2"}

In this case aggregate functions may be used for converting multiple time series into a single time series or into a group of time series. For example, the following query compares the maximum version at env="demo" to the minimum version at env="live" per every time series group with identical job labels:

max(app_version{env="demo"}) by (job)
  !=
min(app_version{env="live"}) by (job)

Another option is to use group_left() or group_right() modifiers, which instruct to match multiple time series on one side of the operator such as != to a single time series on the other side of the operator. See these docs for details. Note that many-to-many time series matching isn't supported by Prometheus.

P.S. The unneeded labels can be also removed with label_del function if MetricsQL is used. For example:

label_del(app_version{env="demo"}, "env") != label_del(app_version{env="live"}, "env")