How to update metric values in prometheus exporter (golang)

Question

I'm starting to write an own prometheus exporter using golang. I think I got the basics but I don't know what to do exactly to get the value of the metric up to date. Using Set only does it once. It is not changing on runtime.

What I have so far:

package main

import (
  "log"
  "net/http"

  "github.com/prometheus/client_golang/prometheus"
  "github.com/prometheus/client_golang/prometheus/promhttp"

  "time"
  "io/ioutil"
  "github.com/tidwall/gjson"
  "strconv"
)

var (
  sidekiqProcessed = setGaugeMetric("sidekiq_processed", "Sidekiq Processed", "lable", "lablevalue")
)

func setGaugeMetric(name string, help string, label string, labelvalue string) (prometheusGauge prometheus.Gauge) {
  var (
    gaugeMetric = prometheus.NewGauge(prometheus.GaugeOpts{
      Name: name,
      Help: help,
      ConstLabels: prometheus.Labels{label: labelvalue},
    })
  )

  return gaugeMetric
}

func getSidekiqProcessed() (sidekiq float64) {
  body := getContent("http://example.com/sidekiq/stats")
  processed := gjson.Get(body, "sidekiq.processed")
  conv, err := strconv.ParseFloat(processed.String(), 64)
  if err != nil {
    log.Fatal(err)
  }
  return conv
}

func getContent(url string) (body string) {
  httpClient := &http.Client{Timeout: 10 * time.Second}
  res, err := httpClient.Get(url)

  if err != nil {
    log.Fatal(err)
  }

  content, err := ioutil.ReadAll(res.Body)
  res.Body.Close()

  if err != nil {
    log.Fatal(err)
  }

  return string(content)
}

func init() {
  prometheus.MustRegister(sidekiqProcessed)
}

func main() {
  sidekiqProcessed.Set(getSidekiqProcessed())

  // The Handler function provides a default handler to expose metrics
  // via an HTTP server. "/metrics" is the usual endpoint for that.
  http.Handle("/metrics", promhttp.Handler())
  log.Fatal(http.ListenAndServe(":8080", nil))
}

Read something about Collector but have no clue how to implement it. Can somebody help me to complete/correct my code so that the value of the metric also updates at runtime?

Answer 1

Here is an example of custom collector (from https://www.robustperception.io/setting-a-prometheus-counter):

package main

import "github.com/prometheus/client_golang/prometheus"

type MyCollector struct {
  counterDesc *prometheus.Desc
}

func (c *MyCollector) Describe(ch chan<- *prometheus.Desc) {
  ch <- c.counterDesc
}

func (c *MyCollector) Collect(ch chan<- prometheus.Metric) {
  value := 1.0 // Your code to fetch the counter value goes here.
  ch <- prometheus.MustNewConstMetric(
    c.counterDesc,
    prometheus.CounterValue,
    value,
  )
}

func NewMyCollector() *MyCollector {
  return &MyCollector{
    counterDesc: prometheus.NewDesc("my_counter_total", "Help string", nil, nil),
  }
}

// To hook in the collector: prometheus.MustRegister(NewMyCollector())

Answer 2

You probably want to implement a collector instead, and run the http request when the Prometheus server scrapes. See the best practices.

When implementing the collector for your exporter, you should never use the usual direct instrumentation approach and then update the metrics on each scrape.

Rather create new metrics each time. In Go this is done with MustNewConstMetric in your Update() method. For Python see https://github.com/prometheus/client_python#custom-collectors and for Java generate a List in your collect method, see StandardExports.java for an example.

Answer 3

The github.com/prometheus/client_golang library may be non-trivial to use when writing Prometheus exporters in Go. Try https://pkg.go.dev/github.com/VictoriaMetrics/metrics library instead. It is much easier to use in general. See the following code as an example, which allows dynamically updating sidekiq_processed metric with the given label:

import (
  "fmt"

  "github.com/VictoriaMetrics/metrics"
)

// UpdateSidekiqProcessed updates `sidekiq_processed{label="<labelValue>"}` metric to the given value
func UpdateSidekiqProcessed(labelValue string, value float64) {
  metricName := fmt.Sprintf("sidekiq_processed{label=%q}", labelValue)
  metrics.GetOrCreateFloatCounter(metricName).Set(value)
}