prometheus rate on series by regex

Question

I am using the following query to get some metrics based on the name:

{__name__=~"bus_listener.+_processed"}

There are multiple metrics that match this name and multiple apps are publishing these metrics.

I am trying to calculate a rate on this, with:

rate({__name__=~"bus_listener.+_processed"}[5m])

But this gives me the following error:

vector cannot contain metrics with the same labelset

I cannot use recording metrics, I only have access to grafana, which reads metrics from prometheus.

How can I get this rate using a regex?

Answer 1

Sounds like you have multiple metrics with the same labels (except for __name__). rate() keeps all labels except __name__, but it drops __name__ to avoid any confusion. Meaning that if you have two time series like:

bus_listener_foo_processed{job="a_job"} 1
bus_listener_bar_processed{job="a_job"} 2

putting them through rate() will result in two time series both with the same labelset:

{job="a_job"} 0.1
{job="a_job"} 0.2

In theory you could duplicate the __name__ label as some other label by using label_replace() first and applying rate() on the result of that, resulting in different labelsets for each original timeseries. However, since you can only compute rate() directly on a timeseries (not the output of another function) you can only do this by using subqueries, which is both heavyweight and slower than it would otherwise be:

rate(label_replace({__name__=~"bus_listener.+_processed"}, "old_name", "$1", "__name__", "(.+)")[5m:1m])

(Optionally replacing 1m with something close to your scrape interval, so there's as little aliasing going on as possible.)

But ideally, if you do have access to the Prometheus configuration (which doesn't seem likely, since you say you can't use recording rules) you should use metric relabeling at ingestion time to extract the various bits of the metric name into separate labels, so you won't have to jump through hoops later on. Or have the service that exports the original metrics use labels instead of concatenating them into the metric name.

Answer 2

I understand that you have multiple "bus_listeners" that report the _processed metrics. Best way would be to have such metrics conform to Prometheus data model and have bus_listener as a label in metric instead of havint it embedded in metric name. That would require changing the application that emits these metrics.

If modifying application is not feasible then you can create conforming new metrics using recording rules.

If you create recording rule similar to:

 - record: processed_count
   expr: label_replace({__name__=~"bus_listener.+_processed", "bus_listener", "$1", "__name__", "bus_listener(.+)_processed")

then you can query new metric just fine:

  rate(processed_count[5m])

As you didn't provide exact metric names this rule may require some adjustments

Answer 3

While this task can be solved by copying metric name from __name__ label to other label with label_replace() function as explained in this answer, this solution isn't ideal because it needs to use subqueries.

MetricsQL provides better solution for this task - keep_metric_names modifier. This modifier instructs the function to leave metric names. It can be applied to any PromQL and MetricsQL function, which removes metric names by default. For example, the following query should leave the original metric names after applying rate() function:

rate({__name__=~"bus_listener.+_processed"}[5m]) keep_metric_names

This solution works faster than the solution with label_replace(), since it doesn't use subqueries.