'prometheus dynamic metrics_path

Prometheus allows me to dynamically load targets with file_sd_config from a .json file like this

#prometheus.yaml
- job_name: 'kube-metrics'
  file_sd_configs:
  - files:
    - 'targets.json'

[
  {
    "labels": {
      "job": "kube-metrics"
    },
    "targets": [
      "http://node1:8080",
      "http://node2:8080"
    ]
  }
]

However my targets differ in the metrics_path and not the host (I want to scrape metrics for every kubernetes node on <kube-api-server>/api/v1/nodes/<node-name>/proxy/metrics/cadvisor) but I can only set the metrics_path at the job level and not per target. Is this even achievable with prometheus or do I have to write my own code to scrape all these metrics and export them at a single target. Also I couldn't find a list of all supported auto discovery mechanisms, did I miss something in the docs?

kubernetesprometheusservice-discovery


Solution 1:[1]

You can use relabel_config in Prometheus config to change __metrics_path__ label config.

The principe is to provide the metrics path in your targets under the form host:port/path/of/metrics (note: drop the http://, it is in scheme parameter of scrape_config)

[
  {
    "targets": [
      "node1:8080/first-metrics",
      "node2:8080/second-metrics"
    ]
  }
]

And then replace the related meta-labels with the parts

- job_name: 'kube-metrics'
  file_sd_configs:
  - files:
    - 'targets.json'
  relabel_configs:
    - source_labels: [__address__]
      regex:  '[^/]+(/.*)'            # capture '/...' part
      target_label: __metrics_path__  # change metrics path
    - source_labels: [__address__]
      regex:  '([^/]+)/.*'            # capture host:port
      target_label: __address__       # change target

You can reuse this method on any label known at configuration time to modify the config of the scrape.

On Prometheus, use the service discovery page to check your config has been correctly modified.

The official list of service discovery is in the configuration documentation: look for the *_sd_config in the index.

Solution 2:[2]

There is a slightly more elegant way of doing this, that doesn't involve manipulating the address. You can have a label with your target that can be the source of your "relabel" action, something like this:

- labels:
    __meta_discovery_path: '/first-metrics'
  targets:
  - 'node1:8080'
- labels:
    __meta_discovery_path: '/second-metrics'
  targets:
  - 'node2:8080'

And then your relabel would simply be:

  relabel_configs:
    - source_labels: [__meta_discovery_path]
      target_label: __metrics_path__  # change metrics path

Since the label being used starts with "__", it will be stripped from the metrics being pulled, making it nice and clean.

Solution 3:[3]

I wanted something like the below endpoints and achieved with this config

192.168.1.1:80/metrics/my-path1
192.168.1.2:80/metrics/my-path1

This will replace $1 with _new_path value.

- job_name: prometheus_dynamic_metrics_path
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  relabel_configs:
  - source_labels: [_new_path]
    separator: ;
    regex: (.*)
    target_label: __metrics_path__
    replacement: /metrics/$1
    action: replace
  static_configs:
  - targets:
    - 192.168.1.1:80
    labels:
      _new_path: my-path1
  - targets:
    - 192.168.1.2:80
    labels:
      _new_path: my-path2

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Michael Doubez
Solution 2 tomo
Solution 3 Milad Jahandideh

Related Questions