Understand Prometheus Metrics Pulling

Question

I'm still not quite understand the Prometheus Metrics Pulling after reading and trying it.

Say I have a telegraf agent that sends metrics over to Prometheus every 5 seconds.

• Prometheus should be configured to pull every 5 seconds right? But what if there are several seconds gap between the push and pull (ok that push and pull not happen at the same time)? What if Prometheus is configured to pull every 7 seconds?
• What if Prometheus is configured to pull every 15 seconds? Only one out 3 push get pulled? Will telegraf agent be fussy about that?
• What if Prometheus is configured to pull every 30 or even 60 seconds? Is the pulled value of that point of time, or average over 30 / 60 seconds?

Lastly, can Prometheus pulling interval be changed during run-time? I want to reduce the pulling interval to every 30 or even 60 seconds at night.

Answer 1

I don't know what kind of exporter you are using, generally, Prometheus metrics exporter is an HTTP server serving at certain endpoint (most cases /metrics).

1. So, when you set scrape_interval=x while configuring Prometheus, it will make a GET request at every x second at the targeted endpoint and store those time-series metrics.

2. If you want to monitor some events which last less than the time mention in scrape_interval, you might miss those events. There is something named prometheus pushgateway to solve this problem.

3. Generally, metrics exporters don't perform any operation over time-series data, you will receive the data of that moment.

Prometheus can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied. A configuration reload is triggered by sending a SIGHUP to the Prometheus process or sending a HTTP POST request to the /-/reload endpoint (when the --web.enable-lifecycle flag is enabled). This will also reload any configured rule files.

Prometheus Configuration doc

Answer 2

Say I have a telegraf agent that sends metrics over to Prometheus every 5 seconds.

Telegraf's Prometheus output plugin does not, unlike the InfluxDB output plugin, push the metrics to the target, but creates a webserver serving the /metrics (default) endpoint.

If you want to use push instead of pull you can use the Pushgateway. The data which is pushed to the Pushgateway (via HTTP POST or PUT) will be available at an endpoint of the Pushgateway, which can be scraped by Prometheus. But note that the Pushgateway should only be used for certain cases, see here.

Telegraf offers some HTTP Output plugin, so you could (theoretically) use Telegraf to push metrics to the Pushgateway. But in this case you should not use Telegraf's Prometheus output plugin additionally.

Answer 3

Prometheus pulls metrics from the configured targets at a regular interval (aka scrape_interval) according to the provided scrape configs. Unlike InfluxDB, Prometheus doesn't accept metrics pushed to it from other services. See this article, which explains why Prometheus supports pull model instead of push model for data collection.

If you still need to push metrics to Prometheus-like system, then take a look at VictoriaMetrics. It supports both pull and push protocols for data ingestion, including InfluxDB line protocol, so Telegraf can be configured to push metrics directly to VictoriaMetrics. See these docs.