'How to monitor the amount of messages in a Kafka topic per day?
I have a Kafka cluster with a topic that receives thousands of messages a day and I want to see how many messages went in the topic per date. I'm using JMX exporter to get some metrics and I can currently see the rate of messages going in the topic, but have no idea how to address this other issue.
This is the formula I use to get the amount of messages per second:
sum without(instance)(rate(kafka_server_brokertopicmetrics_messagesin_total{job="kafka",topic!=""}[5m]))
Solution 1:[1]
Not sure what kind of restrictions you have on which tools you can and can't use. If you're willing to use the Java client, you could create a KafkaConsumer (without a group.id), then assign to it all partitions of the topic you're interested in and then use the offsetsForTimes method twice (once for the beginning of the day in question and once for the end). This will give you the offsets for each partition in the topic at these two points in time and you can just take their difference.
Solution 2:[2]
sum without(instance)(increase(kafka_server_brokertopicmetrics_messagesin_total{topic=~"$topic"}[$__range]))
And you can use "yesterday", "today", "today so far" or a specific date from the dropdown range.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | KWer |
| Solution 2 | Siphalor |