'How can I add a label to the cadvisor and node-exporter metrics?

My node-exporter metrics are something like:

process_cpu_seconds_total{instance="10.1.1.1:8080",job="node_info"}
process_cpu_seconds_total{instance="10.1.1.2:8080",job="node_info"}
process_cpu_seconds_total{instance="10.1.1.15:8080",job="node_info"}

The cadvisor ones:

container_memory_usage_bytes{id="<id>",image="<image>",instance="10.1.1.1:8080",job="docker_info",name="<container name>"}
container_memory_usage_bytes{id="<id>",image="<image>",instance="10.1.1.3:8080",job="docker_info",name="<container name>"}
container_memory_usage_bytes{id="<id>",image="<image>",instance="10.1.1.16:8080",job="docker_info",name="<container name>"}

I want to add a label such as machine_name, something like this:

process_cpu_seconds_total{machine_name="cool_machine",instance="10.1.1.1:8080",job="node_info"}
container_memory_usage_bytes{machine_name="cool_machine",id="<id>",image="<image>",instance="10.1.1.1:8080",job="docker_info",name="<container name>"}

When I try to filter by machine I need to deal with the IP (10.1.1.1), and that is not very user friendly. I wanted to configure node-exporter and cadvisor to add a label to all metrics, this way I can identify the machine no matter what is the IP they have now.

By the way, changing the DNS so the machine answers in another address is not much of an option for me.

My prometheus config is something like:

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'machines_monitor'
scrape_configs:
  - job_name: 'node_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.2:8080
          - 10.1.1.15:8080
  - job_name: 'docker_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.3:8080
          - 10.1.1.16:8080

I can create a scrape_configs for machine and start filtering by that but I don't know if that is a good idea, maybe a performance issue with Prometheus.

I'm trying to add labels to the metrics, but I'm other approaches to help to identify the machines are very welcome.

prometheusgrafanaprometheus-node-exportercadvisor


Solution 1:[1]

You could try the following:

scrape_configs:
  - job_name: 'node_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.2:8080
          - 10.1.1.15:8080
    relabel_configs:
      - source_labels: [__address__]
        regex: '10\.1\.1\.1.+'
        replacement: cool_machine_1
        target_label: machine_name
      - source_labels: [__address__]
        regex: '10\.1\.1\.2.+'
        replacement: cool_machine_2
        target_label: machine_name
      ...

Solution 2:[2]

I have found the solution, I can use Prometheus metric_relabel_configs for that, my config would be something like this:

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'machines_monitor'
scrape_configs:
  - job_name: 'node_info'
    static_configs:
      - targets:
          - 10.1.1.1:8080
          - 10.1.1.2:8080
          - 10.1.1.15:8080
    metric_relabel_configs:
      - source_labels:
          - instance
        target_label: instance
        regex: '10\.1\.1\.1(.*)'
        action: replace
        replacement: cool_machine

There is a related question: Prometheus create label from metric label

Here we can see some other examples: https://gist.github.com/trastle/1aa205354577ef0b329d4b8cc84c674a

Here is a related post: https://www.robustperception.io/controlling-the-instance-label

Solution 3:[3]

The easiest approach to add labels to targets defined in static_configs section is to use labels section - see these docs. For example, the following config would add machine_name label to the scraped metrics from the given statically defined targets:

scrape_configs:
- job_name: node_info
  static_configs:
  - targets: ['10.1.1.1:9100']
    labels:
      machine_name: cool_machine_1
  - targets: ['10.1.1.2:9100']
    labels:
      machine_name: cool_machine_2
  - targets: ['10.1.1.15:9100']
    labels:
      machine_name: cool_machine_15
- job_name: docker_info
  static_configs:
  - targets: ['10.1.1.1:8080']
    labels:
      machine_name: cool_machine_1
  - targets: ['10.1.1.3:8080']
    labels:
      machine_name: cool_machine_3
  - targets: ['10.1.1.16:8080']
    labels:
      machine_name: cool_machine_16

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Marcelo Ávila de Oliveira
Solution 2 Rodolfo
Solution 3 valyala

Related Questions