Category "apache-kafka"

How to write a KafkaAvro Serde for GenericData.Record

I'm using Kafka 0.10.2 and Avro for the serialization of my messages, both for the key and for the value data. Now I would like to use Kafka Streams but I'm stu

Confluent Maven repository not working?

I need to use the Confluent kafka-avro-serializer Maven artifact. From the official guide I should add this repository to my Maven pom <repository> <

kafka deployment on strimzi

i'm trying to deploy kafka with strimzi, problem is, its exposing kafka brokers as load balancers and assigning them an external IP. i want kafka brokers to be

only one client consumed all message in kafka consumer group

There is 3 (node.js + kafka-node) client Following the tutorial, I made topic 'quickstart-events'. Topic: quickstart-events TopicId: oZC9g7FvTiOm-QIuPpyo

Why retention.ms of Kaka Streams repartition topic is set to -1 by default? Isn't this infinitely retain messages in repartition topic?

I think it's related to the below links, but I don't understand. https://issues.apache.org/jira/browse/KAFKA-6535 https://issues.apache.org/jira/browse/KAFKA-61

Kafka producer Interceptor

I am trying add a Interceptor to do validation for the messages published by the producer to Kafka topic. I need to do few validations in addition to Schema val

Kafka Error while creating ACLs NoAuth for /kafka-acl/TransactionalId

I am trying to setup my very first Kafka cluster using confluent 6.0.1 community edition. I have three zookeeper and three kafka nodes. Three server nodes are:

Kafka cluster security for IOT

I am new to the Kafka and want to deploy Kafka Production cluster for IOT. We will be receiving messages from Raspberry Pi over the internet to our Kafka cluste

Kafka, new storage

I'm trying to add new storage for Kafka, here is what I have already done: Add, prepare and mount storage under Linux OS Add new storage in Kafka Broker: log

can a kafka consumer filter messages before polling all of them from a topic?

It was said that consumers can only read the whole topic. No luck doing evaluations on brokers to filter messages. It implies that we have to consume/receive a

Jackson/Kafka LocalDateTime serialization not working properly

We have a very strange/intermittent issue with kafka and jackson. We have a dto defined which has 4 date field startdate, enddate and a embedded metatdata class

-bash: kafka-server-start.sh: command not found

After I installed kafka through homebrew, and my current version is 0.9.0.1. I was following the steps in this post. At step 3, when I run following command in

Flink Missing Events With Windowed Processor(Event Time Windows) and Kafka Source

We have a Streaming Job that has 20 separate pipelines, with each pipeline having one/many Kafka topic sources and with some pipelines having Windowed Processor

Docker compose create kafka topics

Problem: Cannot create topics from docker-compose. I need to create kafka topics before I run a system under test. Planning to use it as a part of the pipeline,

Debezium mySql connecter not able to capture insert and updater operation

I am trying to implemnt CDC piline with Debezium mysql connecter and kafkal But Source connecter not able to pusblish event for insert and update operation in t

Configure zookeeper for kafka

I want to install kafka in my centos 6.5 machine. In kafka installation tutorial, I came to know that it needs zookeeper to run. I have already install hbase wh

Multiple postgres connections with debezium connector

When I create a kafka connect connector with the debezium connector, it results in four database connections. Three of them remain idle, while one works as the

How do I avoid data loss when using Kafka over Kubernetes and one of the nodes fail?

My application runs over a Kubernetes cluster of 3 nodes and uses Kafka to stream data. I am trying to check my system's ability to recover from node failure, s

The input line is too long when starting kafka

I'm trying to run Kafka message queue on Windows. I'm usin this tutorial - https://dzone.com/articles/running-apache-kafka-on-windows-os When i try to run it

Using Kafka as a (CQRS) Eventstore. Good idea?

Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS, eventstore. One of the main points that Ka