Grafana: combining two queries from two prometheus exporters

Question

I have two exporters for feeding data into prometheus - the node exporter and the elasticsearch exporter. I'm trying to combine sources from both exporters into one query, but unfortunately get "No data points" in the graph.

Each of the series successfully shows data:

1. elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"}

2. node_memory_MemTotal{name=~"$node"}

This is the result when I try to subtract the two series from one another:

3. node_memory_MemTotal{name=~"$node"} - elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"}

What am I missing here?

Thanks.

Answer 1

The subtraction you are trying here is more complex than it reads in the beginning. On both sides of the - operator are queries that can result in one or more time series. So the operation requested works as follows: Execute the query on the left hand side and get a result of one or more time series. A time series means a unique combination of a metric and all its labels and their values. Then a second query for your right hand side is executed which also results in one or more time series. Now to calculate the results, only those combinations with matching label combinations are used.

For your example this means that the metrics from node_exporter and from the elasticsearch_exporter have different label names (or even only different values for the labels). When there are no combinations that exist on both sides, you will see the empty result. For details on how operators are applied, please see the prometheus docs.

To solve your problem, you could do the following:

• Check the metrics of both left and right side on their own
• Evaluate if there are additional labels that could be ignored
• See if there is a good label to match on (e.g. instance / node / hostname)
• Use the ignoring(a,b,c) on the required side(s) to drop superfluous dimensions, e.g. the job

Answer 2

Try the following query:

node_memory_MemTotal{name=~"$node"}
  - on(name)
sum(elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"}) by (name)

It works in the following way:

1. It selects all the time series matching the node_memory_MemTotal{name=~"$node"} time series selector.

2. It selects all the time series matching the elasticsearch_jvm_memory_max_bytes{cluster="$cluster", name=~"$node"} selector.

3. It groups time series found at step 2 by name label value and sums time series in each group with sum() aggregate function. The end result of the sum(...) by (name) is per-name sums.

4. It finds pairs of time series with identical name label value from the step 1 and step 3 and calculates the difference between the first and the second time series in each pair. The on(name) modifier is used for limiting the set of labels, which are used for finding time series pairs with matching labels. See more details about this process here.