'Can I find the timestamp of the last value change
Given a time series, is it possible to find the timestamp of the last value change?
For example, a cron job runs every night. It first runs an rsync job, and later computes the size of the target directory. The actual metric value is not all that significant, but I would like to verify that it actually changes every night.
Is it possible to create a query that return a scalar value with the timestamp of the last value change?
Solution 1:[1]
No, you can't do that.
For many use cases, there are workarounds. For mine, there isn't.
Solution 2:[2]
The standard way to handle this is to push a metric whose value is the unix timestamp that the batch jobs succeeded at. You can then alert on that value being recent enough.
https://www.robustperception.io/monitoring-batch-jobs-in-python/ has a Python example.
Solution 3:[3]
Prometheus provides changes function, which can be used for determining the number of changes for time series samples over the given lookbehind window in square brackets. For example, the following query returns the number of sample value changes for time series with the name metric_name during the last 5 minutes:
changes(metric_name[5m])
Zero changes can be filtered out by adding > 0 to the query. The following query returns non-empty result only if there was at least a single sample value change during the last 5 minutes:
changes(metric_name[5m]) > 0
Unfortunately, Prometheus doesn't provide the functionality for returning the timestamp for the last non-empty value :(
If you still need to get the timestamp for the last value change, then VictoriaMetrics may come to rescue with tlast_over_time function. The following MetricsQL query returns the timestamp of the last value change for metric_name with 1 minute precision if the change was made during the last 24 hours:
tlast_over_time(changes(metric_name[5m])[24h:1m])
This query uses subquery feature.
P.S. VictoriaMetrics provides tlast_change_over_time function starting from v1.77.0. This function returns the timestamp in seconds for the last value change on the given lookbehind window in square brackets. For example, the following query is mostly equivalent to the query above. The only difference is that it returns the exact timestamp for the raw sample with the last change:
tlast_change_over_time(metric_name[24h])
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Eirik Lygre |
| Solution 2 | brian-brazil |
| Solution 3 | valyala |