Spring Kafka : Record listener vs Batch listener

Question

With spring-kafka, there is two types of Kafka listeners.

Record Listeners :

@KafkaListener(groupId = "group1", topics = {"my.topic"})
public void listenSingle(String message, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
    /* Process my kafka message */
}

And Batch Listeners :

/*
    Consumer factory is initialized with setBatchListener(true)
*/

@KafkaListener(groupId = "group1", topics = {"my.topic"})
public void listenBatch(List<String> messages, @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) throws Exception {
    messages.forEach({
        /* Process my kafka message */
    });
}

According to the documentation, it doesn't seems to have any impact on the Kafka consumer (which polls multiple messages anyway).

Then I don't understand why should I use the batch listener instead of the other because batch listener have some limitations that the record listener doesn't have (interceptors, offset management, etc...) ?

Maybe I misunderstood something ? What are the benefits of batch listeners ?

Answer 1

In my use case the advantages are not resulting from the processing towards Kafka but in the followed processing in the listener. For example if you have to call a REST API in the message processing of the listener, you're able to do this in a bulk manner with the batch listener. You can pass the whole list to the API. Of course the external API has to support bulk operations, too. Another example could be bulk processing against a database that is accessed while processing the Kafka records.

For example:

@KafkaListener
public void receive(List<AnyPojo> pojo) {
  myPojoRepository.saveAll(pojo);
}

If you do this without batch processing this would result in a new transaction per each record which is way slower than doing it in bulk/batch:

@KafkaListener
public void receive(AnyPojo pojo) {
  myPojoRepository.save(pojo);
}