Dynamic prometheus labels in Gauge

Question

I wonder if/how it's possible to add dynamic labels. I don't know the key or the quantity of the labels which I would like to add to my gauge values.

What I tried

labelsContainers = []string{"node", "container", "pod", "qos", "namespace", "nodepool"}

// Requested resources
requestedContainerCPUCoresGauge = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Namespace: namespace,
        Subsystem: "pod_container_resource_requests",
        Name:      "cpu_cores",
        Help:      "Requested CPU cores in Kubernetes configuration",
    },
    labelsContainers)

for _, containerMetric := range containerMetrics {
    containerLabels := prometheus.Labels{
        "node":      containerMetric.Node,
        "container": containerMetric.Container,
        "qos":       containerMetric.Qos,
        "pod":       containerMetric.Pod,
        "namespace": containerMetric.Namespace,
    }

    for key, value := range containerMetric.NodeLabels {
        containerLabels[key] = value
    }

    requestedContainerCPUCoresGauge.With(containerLabels).Set(containerMetric.RequestedCPUCores)
    requestedContainerRAMBytesGauge.With(containerLabels).Set(containerMetric.RequestedMemoryBytes)
}

The problem

This throws a panic. I assume this is because the prom client expects exactly those labels defined in labelsContainers and doesn't allow further labels. How can I create a gauge which allows additional (unknown) labels?

Answer 1

While the github.com/prometheus/client_golang library doesn't provide an easy way for specifying dynamic labels in metrics, this task can be easily achieved with github.com/VictoriaMetrics/metrics library via GetOrCreate*() functions + the usual fmt.Sprintf() for building dynamic labels. For example:

import (
        "fmt"
        "sort"
        "strings"

        "github.com/VictoriaMetrics/metrics"
)

// UpdateMetric updates metric `name{labels}` to the given value.
func UpdateMetric(name string, labels map[string]string, value float64) {
        // Construct `metric{labels}`
        var labelValues []string
        for k, v := range labels {
                labelValues = append(labelValues, fmt.Sprintf("%s=%q", k, v))
        }
        sort.Strings(labelValues)
        metricName := fmt.Sprintf("%s{%s}", name, strings.Join(labelValues, ","))

        // Update the counter
        metrics.GetOrCreateFloatCounter(metricName).Set(value)
}