Category "apache-kafka"

Akka: persist to Cassandra and publish to Kafka multiple events

I need to store to Cassandra and publish to Kafka multiple events, and call some final handler() only after all events are stored and published. I came across U

How to fix the JAVA Kafka Producer Error "Received invalid metadata error in produce request on partition" and Out of Memory when broker is down

I have been creating a Kafka Producer example using Java. I have been sending normal data which is just "Test" + Integer as value to Kafka. I have use

ERROR Exiting Kafka due to fatal exception (kafka.Kafka$) on Windows - Apache Kafka

I'm getting below error while starting the Kafka-Server on Windows machine. I've downloaded Scala 2.11 - kafka_2.11-2.1.0.tgz from the link: https://kafka.apach

"Kafka Timed out waiting for a node assignment." on MSK

Specs: The serverless Amazon MSK that's in preview. t2.xlarge EC2 instance with Amazon Linux 2 Installed Kafka from https://dlcdn.apache.org/kafka/3.0.0/kafka_2

Why could Kafka warn "partitions have leader brokers without a matching listener"?

I'm trying to get Kafka to work on docker-compose for the first time. The application runs fine without docker. But on docker, I get the error as described belo

Retrieve always latest messages from Kafka on reconnection

I'm writing a piece of code that needs to read hundreds of messages from Kafka each few milliseconds. I'm using C++ and librdkafka. When my program stops and th

how do i add a topic to a running kafka container using docker commands?

I have a kafka container started using following docker run --detach --name kafka -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=192.168.1.89 --env ADVERVTIS

Make kafka consumer group inactive

My biggest question is that do we have a concept in kafka called as consumer group to be inactive/active. I am not sure how to achieve this in my local kafka

Spark streamming take long time read from kafka

I build a cluster use CDH5.14.2, includes 5 nodes, each node has 130G momery and 40 cpu cores. I builded the spark streamming application to read from multiple

Kafka Producer Thread, huge amound of threads even when no message is send

i currently profiled my kafka producer spring boot application and found many "kafka-producer-network-thread"s running (47 in total). Which would never stop run

Apache Kafka - DNS resolution failed for url in bootstrap.servers (Failed to construct kafka producer)

I have a problem to construct kafka producer with hostname. Here's what I did. 1) I set the host.name, port config to server.properties listeners=PLAINTEXT://

Where is the error log file destination for Zookeeper distributed with Kafka?

I'm a bit confused on how to configure where the error log files are written for Zookeeper when running ZK and Kafka on the same node. I understand it uses log

Error sending fetch request (sessionId=1175648978, epoch=189) to node 53: org.apache.kafka.common.errors.DisconnectException

We have a topic with 100 partitions and the load is millions of records per hour. We ran into the problem whenever we deploy a new version of stream-processor

Synchronising transactions between database and Kafka producer

We have a micro-services architecture, with Kafka used as the communication mechanism between the services. Some of the services have their own databases. Say

Unable to connect to Kafka using p12 file in .net core application

I have a .net core application that needs to connect to Kafka using SSL protocol. I have a p12 file which I need to use for authentication with the broker but I

Message loss/missing from same topic, read with different consumer group in Kafka

I have been encountering a weird issue with Kafka and Confluent Sink Connector which I am using in my setup. I have a system where in I have two kafka connect s

Locust Python: Kafka Consumer in a separate thread

I'm using Locust for load testing. I want to register a Kafka consumer in separate thread to measure the time of message processing. Here is what I got now: def

What should I use instead deprecated FlinkKafkaConsumer? Scala Flink

I try to get data from Kafka to Flink, I use FlinkKafkaConsumer but Intellij shows me that it is depricated and also ssh console in Google Cloud shows me this e

Is there a way to manually store Kafka offset so a consumer never misses messages?

Using PHP Laravel Framework to consume kafka messages with the help of the mateusjunges/laravel-kafka laravel package. Is it possible to save the offset by cons

Writing JUnit tests for Kafka Consumer

I have a kafka consumer which is subscribing on a topic. Implementation is working fine. But when trying to implement unit tests for that, there's a problem bec