Time series from Prometheus source: how to set nulls as zero?

Question

Working in Docker Grafana 8.1.5. Using time series graph, I'm plotting a Prometheus Counter source (that has one label) as a time series (by label), and need to fill all null/missing values as zeros.

This is the query applied to the Prometheus counter source, plotting the label code:

my_metric{code!=""}

The graph display works (just need to see the current counter value for each label variant, and the difference within the selected time range), but the new Grafana time series graph is missing an option that the Graph (old) has under Display > Stacking and null value > null value: null as zero, hence it now ends up with broken lines when null values occur.

Unfortunately, I cannot use the Graph (old) chart as I need the legend value difference, which only is available in the new time series graph.

I tried to add or on() vector(0) to the end of the query, but the condition does not get applied to the data series for each label variant, it rather adds a new data series all filled with zeros...

Thanks for any suggestions !

Answer 1

I had this issue as well and I was not able to use only or on() vector(0) as you mentioned because the main query was returning NaN. In my case I had a division by zero.

I could get around of it by first evaluate if the query has a value >= 0 and then use the or on() vector(0). Try something similar to:

((my_metric{code!=""}) >= 0) OR on() vector(0)

Answer 2

The following technique can be used for filling gaps in a single time series returned from the query q:

sum(q) or vector(0)

The q is wrapped into sum() in order to drop all the labels for a time series returned from q, so the time series can be matched with vector(0) time series according to matching rules for or operator. The query without sum(): q or on() vector(0) would return two time series on the graph instead of a single time series: one time series from q with its own set of labels and another time series with zero labels and with zero value where the q time series contains gaps.

Unfortunately Prometheus doesn't provide an easy ability to fill gaps with zeroes if q returns multiple time series with distinct labelsets. Other Prometheus-like solutions such as VictoriaMetrics provide default operator for this case. For example, the following MetricsQL query fills gaps with zeroes for all the time series returned from q:

q default 0