Are kafka acks received in the same order of produced messages

Question

I'm working on a process that collect data from IBM MQ and process it to a kafka topic. To make sure not loosing any message,I need to commit my JMS message only after making sure my message is being sent and received by kafka broker. I don't want to use synchronous kafka producer (waiting on future.get()) because of the performance impact it may have,instead I want to commit my JMS message inside the callback I'm providing my kafka producer. For this to work correctly, I need the garantee that ack will be received in the same order of my produced messages (first ack corresponds to the first message being sent..).

Is my assumption correct?

Answer 1

These are the producer configs you want for ordered Kafka producer messages

enable.idempotence=true
acks=all
max.in.flight.requests.per.connection=5
retries=2147483647

The last two are defaults, so you don't explicitly need to set them.

Details - https://developer.confluent.io/tutorials/message-ordering/kafka.html

However, there will be no guaranteed order of Kafka producer callbacks unless you use a synchronous producer

collect data from IBM MQ and process it to a kafka topic

You can use Kafka Connect for this rather than writing your own producer

Answer 2

We need to talk little about the messages processing guarantee here.

If we are after the exactly-once processing semantics and messaging order, then the consumers must be configured with isolation.level="read_committed" and producers have to be configured with retries=Integer.MAX_VALUE, enable.idempotence=true, and max.in.flight.requests.per.connection=1 per default.

Also, setting max.in.flight.requests.per.connection=1, will guarantee that messages will be written to the broker in the order in which they were sent, even when retries occur.