Category "apache-kafka"

Read from single kafka topic and upsert into multiple oracle tables

I'm using kafka to send the multiple collections data into a single topic using mongo source connector and upsert the data into different oracle tables using jb

kafka and parallel consumer: why order is important into a microservice architecture

I started to dive into kafka ecosystem. I was surprised to find out that by default, each consumer only digests one "event" at a time, sequentially! It's given

Kafka Topic - filter and dispatch messages

Background Our software solution collects data ("events") per customer. Some of the customers (a small fraction ~3%) ask to get this data into "their systems" (

Spring Batch - Resume already started job when master fails

I didn't find an answer to my question so far so I'm giving it a try here. Let's assume a Spring Batch application with remote partitioning. There's one master/

Spring Apache Kafka onFailure Callback of KafkaTemplate not fired on connection error

I'm experimenting a lot with Apache Kafka in a Spring Boot App at the moment. My current goal is to write a REST endpoint that takes in some message payload, wh

Running Kafka Confluent Platform on WSL 2 (Ubuntu Distribution) and Spring application on Windows (Broker may not be available)

I'm running Confluent Platform in wsl 2(Ubuntu Distribution) and I also running a Spring application on Windows but when I send a message with a producer I have

Best way to update/delete records in db based on Apache Kafka topic message

I have an application publishing messages on a Kafka Topic. I've used Confluent's Snow Sync to synchronize the topic with a Snowflake table so when I publish a

Can we create ksql table from topic together with topic data?

I have a db connector, which will detect the timestamp column update on my user table on my database and push it to my kafka topic. Thus, my kafka topic will ke

How to use Kafka with SSL via logback appander?

I use this logback appender to send logs to Kafka: https://github.com/danielwegener/logback-kafka-appender When Kafka was PLAINTEXT everything worked correctly.

Topic authorization failed error while KSQL tries connecting to cluster

I am trying to have my ksqldb-server docker instance up and connect to a remote Kafka cluster, but getting an error. Here are the details docker-compose.yml ---

confluent 5.0.1 upgrade to 7.0.1

Trying to upgrade confluent I have checked upgrade link: https://docs.confluent.io/platform/7.0.1/installation/upgrade.html I found inside it said in the Prepar

Not able to open Control Center UI on localhost

I am trying to understand confluent kafka and do some hands on with it Trying to setup on my local machine . I have followed the instructions in this doc : http

KSQL Rest INSERT not inserting

I have created a KSQL table with the following command: ksql> CREATE TABLE CUSTTABLE (id INT PRIMARY KEY, name VARCHAR, purchase VARCHAR) WITH (KAFKA_TOPIC =

Difference between JDBC Sink Connector vs Redshift Sink Connector

I'm new to Kafka and I would like to know why there are specific Database connectors like Redshift Sink Connector and why we should not go for generic JDBC sink

How to serialize protobuf message with schema id [closed]

Looking for a library or algorithm that implements serialization of a message in protobuf format with schema version which retrieved from conf

kafka - How to read and process messages in a fault tolerant way? [closed]

I am new to Kafka, and I am trying to understand how a Kafka Consumer can consume and process a message from a Kafka topic without losing mess

Kafka Consumer - Point to specific offset in Spring boot Kafka

I'm trying to seek the offset to some specific offset. It seems I need to use seekToTimestamp and there is no seekToOffset or something. I've research a lot but

auto.register.schemas set to false doesn't work as intended

auto.register.schemas=false doesn't work as I expect. If I read the documentation it's suppose to counter the producer to regsiter new schemas. https://docs.c

Access bitnami/kafka running in minikube from local machine

I am trying to access a bitnami/kafka cluster deployed using helm. I need to be able to send messages to kafka from my local machine and then have my pods proc

JSON array serialization issue after spring-cloud-stream update

I use spring-cloud-stream-binder-kafka to connect to the kafka broker I updated spring-cloud-dependencies version to 2021.0.1 I went from this [INFO] +- org.spr