Redpanda seems easy to work with, but how would one process streams in real-time? We have a few thousand IoT devices that send us data every second. We would li
producer sends messages 1, 2, 3, 4 consumer receives messages 1, 2, 3, 4 consumer crashes/disconnects producer sends messages 5, 6, 7 consumer comes back up
I'd like to join data coming in from two Kafka topics ("left" and "right"). Matching records are to be joined using an ID, but if a "left" or a "right" record i
we are using kafka in production and I try to push the adoption and usage of KSQL in the same direction. But I already failed with one simple table-table join.
I'm writing a consumer which listens to a Kafka topic and consumes message whenever message is available. I've tested the logic/code by running Kafka locally an
I'm playing with Kafka engine on ClickHouse cluster. At the moment ClickHouse 22.1 cluster and Kafka are run in Docker. Here are configurations: https://github.
My pipeline is: Kerberized Kafka --> Logstash (hosted on a different server) --> Splunk. Can I replace the Logstash component with Kafka Connect? Could
I am reading this one: Automatic Commit The easiest way to commit offsets is to allow the consumer to do it for you. If you configure enable.auto.commit=t
I use debizium to stream postgresql data to Kafka, and use Java to subscribe Kafka topic. I receive Kafka message and get a JSON string, but
I have a couple of microservices developed using spring boot and each has its own Postgres database. These microservices exchange data with a CDC mechanism prov
I wanted to know why does co-partitioning of two Kstreams in kafka require same number of partitions for both the streams as is given in the documentation in be
I'm trying to implement a custom topology processing step implementing the Processor interface and then adding an instance of my custom processor to the topolog
I try to use Jmeter with plugin "DI-fakfa meter" to load my kafka using SSL. I got the following certificates from VAULT system: -CA -Issuing CA -Private key In
I'm following https://github.com/PacktPublishing/Apache-Kafka-Series---Kafka-Connect-Hands-on-Learning and I've below docker-compose file and using Mac. version
I am using confluent Kafka and schema registry and I cannot figure out a way to have custom schema name in python. From this blog I got how to do it in Java, bu
I am a new beginner in the big data field, I need to make a demo which streams data from Kafka topic using spark stream then make some aggregation and filtering
I'm having Kafka Consumer group of applications (10 instances) written in Java which uses Spring Cloud Stream. Consumer application is deployed in AWS Kubernete
In another similar question, they hint 'install older spark 2.4.5.' EDIT: the solution from above link says 'install spark 2.4.5 and it does have kafkautils. Bu
I'm using Spring Kafka integration and I've my own value generic serializer/deserializer as shown below Serializer: public class KafkaSerializer<T>
I am getting intermittent issues while accessing the kafka service from the Kubernetes pod. org.apache.kafka.common.KafkaException: Failed to construct kafka pr