Adding metric labels in prometheus on the fly

Question

I have a counter metric in prometheus. I want to add lables to it dynamically for example if my request comes http://abc123.com/{p1} ,I want my custom_metric_name to store {statuscode=200, p1=p1Value , host="abc123"} and if request comes http://def123.com/{p2} . I want custom_metric_name to store {statuscode=200, p2=p2Value , host="def123"} but custom_metric_name will be shared metric by both.

I am trying still not able to get answer

Answer 1

You can use relabel_config or metric_relabel_config in your Prometheus config.

It would look like the following:

- source_labels: [request_origin]
  regex: 'http://(\w+)/.*'
  replacement: '${1}'
  target_label: host

See also this article showing usage of relabelling.

Answer 2

Dynamic labels can be easily added to the exported Prometheus metrics when using github.com/VictoriaMetrics/metrics Go package:

import (
  "fmt"
  "github.com/VictoriaMetrics/metrics"
)

// CountRequest increments `requests_total{statuscode="statusCode", host="host", path="path"}` Prometheus counter
func CountRequest(statusCode int, host, path string) {
  s := fmt.Sprintf(`requests_total{statuscode="%d", host=%q, path=%q}`, statusCode, host, path)
  metrics.GetOrCreateCounter(s).Inc()
}