Does Kafka config "retention.bytes" apply to compact topic?

Question

Does "retention.bytes" apply to "compact" topic? The reason why I came here is that in lenses in my current project, I saw the partition bytes is 2GB which is way more large than the configured "rention.bytes" 43MB. Based on the doc, seems like it does not.

cleanup.policy A string that is either "delete" or "compact" or both. This string designates the retention policy to use on old log segments. The default policy ("delete") will discard old segments when their retention time or size limit has been reached. The "compact" setting will enable log compaction on the topic.

retention.bytes This configuration controls the maximum size a partition (which consists of log segments) can grow to before we will discard old log segments to free up space if we are using the "delete" retention policy. By default there is no size limit only a time limit. Since this limit is enforced at the partition level, multiply it by the number of partitions to compute the topic retention in bytes.

In the "cleanup.policy", it is saying deleting is happening when retention size is reached for "delete" topic. "log compaction" will happen for compact topic but without mentioning any condition.

In the "retention.bytes", it only mentions the delete strategy for "delete" topic, not a word for "compact" topic.

If "retention.bytes" does not apply to "compact" topic, does that mean it will always keep a unique key in the topic? If that is the case, say the key space is too large that it consumes huge amount of disk space, how can we avoid this?

Last question: if above is true, can I say for a purely "compact" topic, compared with "delete" and "delete,compact" cleanup policy, there is no point to set "retention.bytes" and "retention.ms" for it?

Answer 1

Compaction will keep all unique keys in the "head" of the topic, yes, with no limit. The tail, unclosed segment (see segment.bytes) will remain un-compacted until the log cleaner compacts it, and then storage usage will grow for new un-compacted segments.

You can use compact.policy=compact,delete to have retention.ms and/or retention.bytes get applied to compacted topics (as the docs say, "by default there is no size limit only a time limit")