'how do i add a topic to a running kafka container using docker commands?

I have a kafka container started using following

docker run --detach --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.168.1.89 --env ADVERVTISED_PORT=9092 --env AUTO.CREATE.TOPICS.ENABLE spotify/kafka

i can use docker logs kafka to see its started.

I then created a simple groovy script client producer to write some entries, however this keeps erroring with

> Sending metadata request {topics=[wills topic]} to node 0
> Error while fetching metadata with correlation id 1 : {wills topic=INVALID_TOPIC_EXCEPTION}
....

I have set the following properties in the client code

    Properties props = new Properties()
    props.put("bootstrap.servers", "192.168.1.89:9092" )   //Assign localhost id and external port (9092 int)
    props.put("acks", "all")                            //Set acknowledgements for producer requests.
    props.put("retries", 0)                             //If the request fails, the producer can automatically retry,
    props.put("batch.size", 16384)                      //Specify buffer size in config
    props.put("linger.ms", 1)                           //Reduce the no of requests less than 0
    props.put("buffer.memory", 33554432)                //The buffer.memory controls the total amount of memory available to the producer for buffering.
    props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
    props.put ("auto.create.topics.enable", true)       //enable auto topic creation

    producer = new org.apache.kafka.clients.producer.KafkaProducer<String, String>(props)

    for(int i = 0; i < 10; i++) {
        producer.send(new ProducerRecord<String, String>(topicName, Integer.toString(i), Integer.toString(i)))
    }
    println("Message sent successfully")
    producer.close()

but that does not not seem to enable auto topic creation. Given that i've spun the kafka server up as docker container - how does one administer that container instance once it is up and running using docker commands as the means to talk to the container

there is a note on the kafka site along the line of running a shell commend "bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test"

but its not easy to see how one edits/interacts with this in container once its up.

how does one administer the kafka container instance in docker once it has been started. otherwise what to you have to add to the docker run command to get topics pre created at startup

Advice greatfully received at this point

Solution 1:^[1]

The error message {wills topic=INVALID_TOPIC_EXCEPTION} is indicating that you are using wills topic as topic name.

A topic name cannot contain whitespaces. Try to rename to wills_topic and it should solve the problem.

This regex describes the legal characters for topic names (check sources):

val legalChars = "[a-zA-Z0-9\\._\\-]"

Use docker exec -it <container-name> <command> to launch the Kafka admin tools. Or just open a bash within your container docker exec -it <container-name> bash (see Docker docs).

Solution 2:^[2]

use docker exec -it [container_id] /opt/kafka_2.11-0.10.1.0/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Solution 3:^[3]

I have tried the above steps but facing some permission problems there is one more solution you can try in mine it worked perfectly

Download Apache Kafka binary file from the official website https://kafka.apache.org/downloads.
Open the terminal go through the path up to the folder. For eg /Users//Downloads/kafka_2.13-3.0.0
And run this command

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic 'topic-name'

If you want to create multiple topics with a single statement only

Create a text file first and save it topics.txt in Kafka folder in a format like ::

Eg:-

topic1:1:1

topic2:1:1

Open terminal up to the downloaded Kafka folder
Run the command with the help of awk

awk -F':' '{ system("./bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --topic=" $1 " --partitions=" $2 " --replication-factor=" $3) }' /topics.txt

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1
Solution 2	Ayeshmantha Perera
Solution 3

'how do i add a topic to a running kafka container using docker commands?

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]

Solution 3:^[3]