Prometheus rule variable?

Question

I have a Prometheus Federator with multiple targets. I need to customize its rules (thresholds) for eventually each target.

Is there a way to set a have a rule that behaves slightly differently with 2 targets ? Using labels are possible on PromQL expressions ? Are there variables on PromQL expressions ?

For example:

alert: Mem
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < $myCustomIntHere

How to have 2 or more thresholds (defined at target level) with a single rule ?

Answer 1

You can define rules like the following example:

alert: Mem

expr: node_memory_MemAvailable_bytes{instance="foo"} / node_memory_MemTotal_bytes{instance="foo"} * 100 < 70

expr: node_memory_MemAvailable_bytes{instance="boo"} / node_memory_MemTotal_bytes{instance="boo"} * 100 < 80

expr: node_memory_MemAvailable_bytes{instance!~"foo|boo"} / node_memory_MemTotal_bytes{instance!~"foo|boo"} * 100 < 90