'Docker kafka on MacOS M1 Issues stuck on configuring

I use macOS M1 Big Sur 11.2.3, but my kafka cannot running well and cannot create/list the topics. I don't know its because the OS or not, but the log for kafka is only like this:

docker-compose logs

list the topics logs

here's my docker compose:

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
    networks:
      - kafka_net
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - 9092:9092
    expose:
      - 29092
    depends_on:
      - zookeeper
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LISTENERS: INSIDE://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 1
    restart: always
    networks:
      - kafka_net
networks:
  kafka_net:
    driver: bridge

I think kafka not yet to running, So i cannot list/create a topics. Do you guys have any idea of this? i already search possibilities about this, but i still don't have the problem solving. Thanks

macosdockerapache-kafkaapple-m1


Solution 1:[1]

For startup kafka in docker on mac m1 i do next things :

  1. Clone kafka repo and build on my macbook pro m1 (./gradlew clean releaseTarGz)
  2. Create docker-compose project dir with next structure
  • root dir
    • kafka_m1
      • kafka (unziped kafka distr)
      • Dockerfile
    • docker-compose.yml
  1. Configure server.properties file (root dir/kafka_m1/kafka/config/server.properties). Main params:
   zookeeper.connect=zookeeper:2181
   listeners=PLAINTEXT://:9092
   advertised.listeners=PLAINTEXT://127.0.0.1:9092
   listener.security.protocol.map=PLAINTEXT:PLAINTEXT
  1. kafka`s Dockerfile (openjdk:15.0.2-jdk was build for m1):
    FROM openjdk:15.0.2-jdk
    
    WORKDIR /
    COPY . /
    
    EXPOSE 9092
    EXPOSE 8092
    EXPOSE 9092
    EXPOSE 10092
    EXPOSE 11092
    EXPOSE 12092
    
    ENTRYPOINT [ "/kafka/bin/kafka-server-start.sh", "/kafka/config/server.properties" ]
  1. docker-compose.yml (i choose zookeeper:3.7.0 because it has arm build):
  version: "2"
  services:
  zookeeper:
    image: zookeeper:3.7.0
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    build: ./kafka_m1
    ports:
      - "127.0.0.1:9092:9092"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  1. Start docker-compose with kafka_m1 image building.

It`s fast receipt for run local kafka in docker

Solution 2:[2]

the docker-compose.yaml file is ok, you just have to execute the following commands into the kafka container:

  1. kafka-topics.sh --zookeeper zookeeper:2181 --topic test --create --partitions 3 --replication-factor 1
  2. kafka-topics.sh --zookeeper zookeeper:2181 --list

btw: you can enter to the kafka container using the command: docker exec -it CONTAINER_ID bash

Solution 3:[3]

Just add platform to your docker-compose.yml file:

platform: linux/amd64

services:
  zookeeper:
    image: wurstmeister/zookeeper
    platform: linux/amd64
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - 2181:2181
    environment:
      ZOO_MY_ID: 1
    networks:
      - kafka_net
    ...

If above option doesn't help then you should try the following:

version: '3'

services:
  zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
  
  kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Igor Fedorov
Solution 2 luisbar
Solution 3

Related Questions