Category "apache-kafka"

How to control memory usage of stream created on ksql cluster

We are using KSQL Version 0.23.1. We are seeing OOM issues on ksql cluster which has persistent stateless stream. Stream is just doing the point filtering on th

Flink - Failed to get metadata for topics exception while consuming from kafka topic

I have setup flink operator and kafka on minikube (both are on different namespaces). I am trying to consume messages from a kafka topic in flink and do some tr

Need to Consume Multiple Kafka Topics

We have created a .NET Worker.cs to consume kafka topics. What should be the best way to consume multiple topics? Our requirement is there are multiple topics,

How to ensure exactly once processing in Kafka for older version 0.10.0

I have a scenario where I need to call an api to the notification service which will send an email to the user. Based on the message that appears to the topic I

Kafka working in Azure Kubernetes cluster, but not locally in minikube

I am using Kafka and facing a problem. Kafka doesn't seem to work locally using minikube. I get this erorr: org.apache.kafka.common.errors.InvalidReplicationFac

Graddle wrapper command gives 'Could not resolve all artifacts for configuration ':classpath''

I tried many of similar solutions - had no luck. I'm using Ubuntu 20.04, following this Confluent tutorial I've managed to get everything running several times

How can I configured jaas.conf in order to Kafka Inbound Endpoint WSO2 EI 6.6.0?

I have WSO2 EI 6.6.0 and I am using the next libs to configure Kafka Inbound Enpoint. dropins: org.apache.synapse.kafka.poll-1.0.10.jar lib: kafka_2.12-1.0.0.

Connect to confluent schema registry SSL configuration - unable to find valid certification path to requested target

How should it be configured the spring-cloud-schema-registry-client to communicate with Confluent Schema Registry on a secured manner using SSL truststore? My c

Half of messages lost when Spark Streaming (4 concurrent drivers) reads from Kafka and writes to MongoDB

I have set up a docker network with: 4 producer containers (each scraping a different forum) that produce to a Kafka container (1 topic for each of the forum) A

Too many org.apache.kafka.common.metrics.stats.* instance been loaded caused OOM issue

we have project using kafka and kafka stream, recently we found OOM issue happen for the project, after analysis the dump file, we found that too many 'org.apac

Embedded Kafka contract tests randomly failing

I am using the spring cloud contrat to test a kafkaListsener of my service. I use EmbeddedKafka together with the spring cloud contract. The listener in my serv

Kafka deployed through istio cannot be provided to the k8s cluster for external access

Kafka deployed through istio cannot be provided to the k8s cluster for external access with the following error Error connecting to node kafka-0.kafka-headless

Kafka org.apache.kafka.common.errors.TimeoutException for some of the records

Getting below error message when producing record to Azure event hub(kafka enabled) Expiring 14 record(s) for eventhubname: 30125 ms has passed since batch cre

No data in the changelog topic for a KSQL table

Step 1. Postgres CDC Data I have a Postgres table which has been set up to use Debezium PostgreSQL Connector to generate CDC. In Kafka, I can see the CDC data i

Read from single kafka topic and upsert into multiple oracle tables

I'm using kafka to send the multiple collections data into a single topic using mongo source connector and upsert the data into different oracle tables using jb

kafka and parallel consumer: why order is important into a microservice architecture

I started to dive into kafka ecosystem. I was surprised to find out that by default, each consumer only digests one "event" at a time, sequentially! It's given

Kafka Topic - filter and dispatch messages

Background Our software solution collects data ("events") per customer. Some of the customers (a small fraction ~3%) ask to get this data into "their systems" (

Spring Batch - Resume already started job when master fails

I didn't find an answer to my question so far so I'm giving it a try here. Let's assume a Spring Batch application with remote partitioning. There's one master/

Spring Apache Kafka onFailure Callback of KafkaTemplate not fired on connection error

I'm experimenting a lot with Apache Kafka in a Spring Boot App at the moment. My current goal is to write a REST endpoint that takes in some message payload, wh

Running Kafka Confluent Platform on WSL 2 (Ubuntu Distribution) and Spring application on Windows (Broker may not be available)

I'm running Confluent Platform in wsl 2(Ubuntu Distribution) and I also running a Spring application on Windows but when I send a message with a producer I have