'Kafka MirrorMaker - from Older version to Newer Version
I want to mirroring from Kafka Source Cluster to Kafka Destination Cluster. Everything is working fine if my both Source and Target Cluster are on the same version (say 0.10.0.0) but its not working when my Source Cluster is on 0.8.2.1 and Target Cluster is on 0.10.0.0.
Does anybody have any idea?
Thanks in advance.
Solution 1:[1]
I'm encountering this same issue. It appears that the mirrormaker is only works if the source and target clusters are the same version.
Solution 2:[2]
I can confirm that this won't work. I tested using MirrorMaker from Confluent Platform 3.0. I was trying to replicate from Kafka 0.8.2.2 to Kafka 0.10.1.1. And I could not get them to work. Worked fine on 0.8.2.2 to 0.8.2.2 and fine on 0.10.1.1 to 0.10.1.1.
Also MirrorMaker does not sync any consumers offsets from old to new. This means any messages consumed in the old cluster will need to consumed again in the new cluster. Unless there is some kind of TTL or your application logic can cater for duplicate messages. But even if your app has logic catering for duplicates if you need to re process millions of messages this would be a waste of resources.
We were trying to setup two live clusters with different versions and using MirrorMaker to sync between them and have some of the older versions of consumers consuming off old cluster and newer versions of consumers consuming off new cluster but this now has proven not working.
Solution 3:[3]
I got it to work by using the mirror tool of the old version to consume from the old version and to produce messages on the new version. Vice versa does not work. That was between a 0.9 and a 2.7 version though.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | user7328120 |
| Solution 2 | Jeff |
| Solution 3 |