'Prometheus scrape config for multiple metric endpoints per pod

We have a Kubernetes Pod which provides multiple metric endpoints (:3093/metrics and :9113/metrics), but it also has an port which doesn't provide any metric (:80).

TL;DR: Is is possible to scrape only the ports 3093 and 9113?

We are using the example configuration for the scrape configuration:

- job_name: 'kubernetes-pods'
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

These endpoints get properly scraped, when enabling scraping with this annotation:

  annotations:
    prometheus.io/scrape: "true"

But this has the issue, that it also scrapes port :80, which it shouldn't.

kubernetesprometheus


Solution 1:[1]

We created an exporter which merges the output of multiple other exporters. It is very alpha, but it works for us, now.

https://github.com/rebuy-de/exporter-merger

Solution 2:[2]

The suggestion outlined here allows scraping multiple endpoints per pod. The idea is to give custom names to container ports, which must be scraped, so these names can be used later during relabeling phase for the discovered targets. For example, the following config adds prom-metrics name to ports 3093 and 9113, which must be scraped by Prometheus. Note that the port 80 has no prom-metrics name, so it won't be scraped.

containers:
- name: custom-app
  image: github.com/user/app:latest
  ports:
  - containerPort: 80
  - containerPort: 3093
    name: prom-metrics
  - containerPort: 9113
    name: prom-metrics

Later the following scrape config can be used for scraping container ports with prom-metrics name:

  - job_name: 'pods-with-prom-metrics'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_container_port_name]
      regex: prom-metrics
      action: keep
    - source_labels: [__address__, __meta_kubernetes_pod_container_port_number]
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_pod_container_name]
      target_label: container
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 svenwltr
Solution 2

Related Questions