'Kafka infinite loop of error "SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation"
I'm having Kafka Consumer group of applications (10 instances) written in Java which uses Spring Cloud Stream. Consumer application is deployed in AWS Kubernetes cluster. Consumer config is using the default values (for ex., max poll interval = 5 mins). Application seems to be working without any issues till if one of the pod gets killed/evicted by node for whatever reason. Once a pod gets evicted, new pod gets added without any issues, but consumer group gets disturbed and goes into infinite loop of rebalancing with below error,
Triggering the followup rebalance scheduled for 0 ms
Request joining group due to: rebalance enforced by user.
SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation.
My expectation was if any pod killed/evicted, new pod gets added to consumer group and a rebalance occurs once and after rebalance everything should work normal. But that's not happening here. Any help is much appreciated.
Update: Above mentioned error occurs every 5 mins (which is the same as max poll interval).
I'm using Kafka - 3.0.1, Spring kafka - 2.8.4 and Spring Cloud Stream - 3.2.1
Please let me know for more information
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|