Kafka, new storage

Question

I'm trying to add new storage for Kafka, here is what I have already done:

1. Add, prepare and mount storage under Linux OS
2. Add new storage in Kafka Broker: log.dirs: /data0/kafka-logs,/data1/kafka-logs
3. Restart Kafka Brokers
4. New directories under /data1/kafka-logs has been created but the size is:

du -csh /data1/kafka-logs/ 156K /data1/kafka-logs/

And the size isn't growing only the old /data0 is used.

What I'm missing? What should I do more to solve this problem? The storage is almost full, and the /data1 is still not used.

Answer 1

When you add a new data storage path for Kafka, the old data does not start automatically rebalancing on this disc. The new storage will be used when you are, for example, create a new topic, or increase the number of partitions in the existing ones.

In that case, if you need to move some data to a new disk, you will need to use manual data rebalancing. For more details see: https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

In your case, to prevent the complete exhaustion of disk space, you can for some time to reduce the lifetime of the topic (retention.ms), otherwise the broker will fall.

Answer 2

I can only guess - but for me it sounds like you did NOT created a new topic or increased the number of partitions of your current topic(s). Kafka will not move the existing partitions into the new log.dir. According to the kafka documentation:

... Each new partition that is created will be placed in the directory which currently has the fewest partitions. - https://kafka.apache.org/08/configuration.html