Kafka Consumer is not receiving Messages on docker

Question

I'm a begginer on kafka as well as docker, I have been doing a course and working with kafka producer and consumer but for some reason it is not working. When I do use of the producer the message are saved in the topic (I have already checked it) but when I try to get the message using the consumer it is not working and I have no idea why.

It had worked previously but not anymore.

The unique difference I have in this case is that I'm using the confluentinc image instead of the bitnami image.

So, if anyone has any idea or solution I would really appreciate it.

I share my compose and an screenshot so you can see it.

version: "3.2"
services:
###############################################################
  zookeeper:
    image: 'confluentinc/cp-zookeeper:latest'
    container_name: zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 2181:2181

###############################################################
  broker:
    image: 'confluentinc/cp-kafka:latest'
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      # Exposes 9092 for external connections to the broker
      # Use kafka:29092 for connections internal on the docker network
      # See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for details
      KAFKA_LISTENERS: "PLAINTEXT://:29092,PLAINTEXT_HOST://:9092"
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_DELETE_TOPIC_ENABLE: "true"

Producer and Consumer

All is running in my local machine.

Answer 1

Take a look at docker-compose logs broker...

You should see a lot of Error processing create topic request CreatableTopic(name='__consumer_offsets', numPartitions=50, replicationFactor=3

Without a valid __consumer_offsets topic, no consumer will be able to run and commit offsets. Similarly, transactions won't work either (which are enabled by default in latest Kafka)

Add these variables and re-create the containers

KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1