'Use conditional operator in Prometheus alert rules to set severity

I would like to use the conditional operator into the Prometheus alert.rules definition to set a specific severity. For instance, if the environment is production, I want to set the severity to critical else another value.

Something like:

- alert: CPU load
  expr: expression_used_to_check_load
  for: 15m
  labels:
     severity: if $labels.env == prd -> severity = critical else something else
  annotations:
     summary: Just a summary
     description: "Just a description"

Solution 1:^[1]

Rather than configuring this in the severity itself, you can configure the receiver of your pages differently based on environment.

- match: {owner: 'MYTEAM'}
    receiver: 'blackhole'
    routes:
    - match: {severity: 'critical', env: 'staging'}
      receiver: 'myteam-warn'
    - match: {severity: 'critical', env: 'prd'}}
      receiver: 'myteam-page'

The different receivers can be configured differently. myteam-warn might just send you an email whereas myteam-page might trigger a page.

Solution 2:^[2]

You could divide the rule in two:

- alert: CPU load
  expr: expression_used_to_check_load and {env=="prd"}
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: Just a summary
    description: "Just a description"
- alert: CPU load
  expr: expression_used_to_check_load and {env!="prd"}
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: Just a summary
    description: "Just a description"

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Stephen Ostermiller
Solution 2	Stephen Ostermiller

'Use conditional operator in Prometheus alert rules to set severity

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]