Micrometer - Prometheus Gauge displays NaN

Question

I am trying to generate Prometheus metrics with using Micrometer.io with Spring Boot 2.0.0.RELEASE.

When I am trying to expose the size of a List as Gauge, it keeps displaying NaN. In the documentation it says that;

It is your responsibility to hold a strong reference to the state object that you are measuring with a Gauge.

I have tried some different ways but I could not solve the problem. Here is my code with some trials.

import io.micrometer.core.instrument.*;
import io.swagger.backend.model.Product;
import io.swagger.backend.service.ProductService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;

@RestController
@RequestMapping("metrics")
public class ExampleController {

    private AtomicInteger atomicInteger = new AtomicInteger();

    private ProductService productService;
    private final Gauge productGauge;

    @Autowired
    public HelloController(ProductService productService,
                           MeterRegistry registry) {

        this.productService = productService;

        createGauge("product_gauge", productService.getProducts(), registry);
    }

    private void createGauge(String metricName, List<Product> products,
                                    MeterRegistry registry) {

        List<Product> products = productService.getProducts();

        // #1
        // this displays product_gauge as NaN
        AtomicInteger n = registry.gauge("product_gauge", new AtomicInteger(0));
        n.set(1);
        n.set(2);

        // #2
        // this also displays product_gauge as NaN
        Gauge
            .builder("product_gauge", products, List::size)
            .register(registry);

        // #3
        // this displays also NaN
        testListReference = Arrays.asList(1, 2);
        Gauge
            .builder("random_gauge", testListReference, List::size)
            .register(registry);

        // #4
        // this also displays NaN
        AtomicInteger currentHttpRequests = registry.gauge("current.http.requests", new AtomicInteger(0));
    }

    @GetMapping(path = "/product/decrement")
    public Counter decrementAndGetProductCounter() {
        // decrement the gague by one
    }
}

Is there anyone who can help with this issue? Any help would be appreciated.

Answer 1

In all cases, you must hold a strong reference to the observed instance. When your createGauge() method is exited, all function stack allocated references are eligible for garbage collection.

For #1, pass your atomicInteger field like this: registry.gauge("my_ai", atomicInteger);. Then increment/decrement as you wish. Whenever micrometer needs to query it, it will as long as it finds the reference.

For #2, pass your productService field and a lambda. Basically whenever the gauge is queried, it will call that lambda with the provided object: registry.gauge("product_gauge", productService, productService -> productService.getProducts().size());

(No guarantee regarding syntax errors.)

Answer 2

I wasn't able to use @panser solution 'cause I'm using gauges with labels. My solution involved the creation of com.google.common.util.concurrent.AtomicDouble cache with io.micrometer.core.instrument.Tag's key and values as map key, heres goes:

    private static final Map<String, AtomicDouble> GAUGE_CACHE = new HashMap<>();

    public void handleGauge(String name, List<Tag> tags, double value) {
        String gaugeKey = this.gaugeKey(name, tags);
        if (!GAUGE_CACHE.containsKey(gaugeKey)) {
            GAUGE_CACHE.put(gaugeKey, new AtomicDouble());
        }
        Objects.requireNonNull(this.registry.gauge(name, tags, GAUGE_CACHE.get(gaugeKey))).set(value);
    }

    private String gaugeKey(String name, List<Tag> tags) {
        return name + ":" + tags.stream().map(tag -> tag.getKey() + tag.getValue()).collect(Collectors.joining(""));
    }

That worked pretty well for my needs, hopefully help other people.

Answer 3

I had the same issue with Micrometer.io gauges when I used your method #1 meterRegistry.gauge("myGauge", new AtomicDouble()). I am using Scala by the way. I noticed that after I created about 50 gauges, the new gauges after that displayed NaN.

Instead I used:

val atomicDouble = new AtomicDouble()
Gauge
  .builder("myGauge", atomicDouble, new AtomicDoubleToDoubleFunction)
  .strongReference(true)
  .register(meterRegistry)

with

class AtomicDoubleToDoubleFunction extends ToDoubleFunction[AtomicDouble] {
  override def applyAsDouble(value: AtomicDouble): Double = value.doubleValue()
}

This fixed the NaN issue, and all of my gauges appear correctly. I found the .strongReference(true) example from https://www.codota.com/code/java/classes/io.micrometer.core.instrument.Gauge .

Answer 4

my example for gauge using

private final AtomicLong countTryUsers = new AtomicLong(0);
Metrics.gauge("app.countTry", countTryUsers);

public void updateCountTryUsers(Long countTryUsersDb){
   countTryUsers.set(countTryUsersDb);
}

so I register app.countTry just once, and then just update AtomicLong countTryUsers over custom method updateCountTryUsers()

Answer 5

I was getting the same issue, in spring boot with actuator and prometheus, where I was trying to create a metric for Map size. When I register the map with the guageMapSize, for some time it was showing the correct value, and after that it just stopped updating, and continuously showed NAN, as the metric for map with key already present, it was not updating.

To resolve this we can remove the existing metric and register the new one with the updated value. This worked for me. Hope it works for you.

Create a method, removeMicroMeterGuage(); use it to remove a gauge, and then add new gauge with the updated value.

removeMicroMeterGuage(<keyName>);
registry.gaugeMapSize(<KeyName>, null, map);



 public void removeMicroMeterGuage(String guageName) {
        List<Meter> meterList = registry.getMeters();
        for(Meter meter : meterList){
            meter.getId().getName();
            if(meter.getId().getName().equalsIgnoreCase(guageName)){
                System.out.println(guageName+" guage removed");
                registry.remove(meter);
            }
        }
    }