'Grafana alert when change between two data points is high
I have a Grafana graph with data coming in from Prometheus. I'm trying to get an alert set up so that if the change over one minute (or so, I'm flexible) is over 60, an email notification goes out. I can't seem to find anything in the documentation that suggests a way to do this, or if it's possible at all. Any ideas?
Solution 1:[1]
If you're just interested in the delta between points you can can plot the derivative() of your series, and alert on that.
Solution 2:[2]
The Prometheus sum_over_time or rate functions are probably what you are looking for. There is also a deriv function.
The Grafana demo site has some Prometheus query examples.
Solution 3:[3]
You need to use delta function. For example, the following query returns non-empty result (aka alert) when the metric foo increased by more than 60 during the last minute (see [1m] in square brackets):
delta(foo[1m]) > 60
It is possible to use any supported time duration in square brackets (aka lookbehind window) for caclulating the metric delta on different intervals. See these docs.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | AussieDan |
| Solution 2 | Daniel Lee |
| Solution 3 | valyala |