Why retention.ms of Kaka Streams repartition topic is set to -1 by default? Isn't this infinitely retain messages in repartition topic?

Question

I think it's related to the below links, but I don't understand.

• https://issues.apache.org/jira/browse/KAFKA-6535
• https://issues.apache.org/jira/browse/KAFKA-6150
• Kafka Streams deleting consumed repartition records, to reduce disk usage

It's possible to provide topic configurations like "retention.ms", "cleanup.policy" for kafka streams internal topics like *-changelog topics to delete useless logs.

But when it comes to internal topics like *-repartition topics, it's not possible to provide topic configuration values, even though the default "retention.ms" for repartition topic is "-1" which means infinite retention. How can I delete or manage repartition topics? Otherwise the repartition topic's size is going to be too large and disk malfunction problems might occur.

How can I manage repartition topics? What is purgeData? Couldn't find any related explanations on the documentation.

Answer 1

Fact

• retention.ms for the repartition topics is -1 by default and there's no way to override this value in kafka-streams client code.

What I misunderstood

• Size of the repartition topic would be increasing infinitely since the retentions.ms for the repartition topics is -1.

Fix misunderstanding

• There's a method called maybeCommit in the StreamThread class.
• maybeCommit method is called iteratively inside the loop that handles stream records.
• Inside the maybeCommit method (version 2.7.1), there's a comment like below.

  try to purge the committed records for repartition topics if possible

  • https://github.com/apache/kafka/blob/2.7.1/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java#L923-L926
• Based on this, what I understand is that when the record in the repartition topics is streamed down to the changelog topic, then the records already sent are purged periodically.
• Therefore, there's no need to clear or manage retention.ms for the repartition topics.

Reference

• https://issues.apache.org/jira/browse/KAFKA-6150

Please leave a comment or correct this if I'm wrong.