'Debezium MySQL not capturing INSERT operations
I am running MySQL in a single instance and Debezium as Docker-containers on my local machine via docker-compose
mysql:
image: mysql:latest
container_name: mysql
command:
--server-id=12345
--log-bin=mysql-bin
--binlog-format=ROW
--binlog-row-image=full
networks:
- tutorial
environment:
MYSQL_USER: foo
MYSQL_PASSWORD: bar
MYSQL_ROOT_PASSWORD: foobar
volumes:
- db_data:/var/lib/mysql
connect:
image: quay.io/debezium/connect:1.9
container_name: connect
ports:
- "8083:8083"
networks:
- tutorial
environment:
BOOTSTRAP_SERVERS: <broker>
CONFIG_STORAGE_TOPIC: connect-configs
OFFSET_STORAGE_TOPIC: connect-offsets
STATUS_STORAGE_TOPIC: connect-statuses
CONNECT_SECURITY_PROTOCOL: SASL_SSL
CONNECT_SASL_MECHANISM: SCRAM-SHA-256
CONNECT_SASL_JAAS_CONFIG: <config>
volumes:
db_data:
networks:
tutorial:
The Kafka broker is on Upstash where I have created a cluster with the following topics:
- connect-offsets
- connect-statuses
- connect-configs
- schema-changes.testdb
- brokername.testdb.planes
My connector config looks as follows
{
"name": "tutorial-connector",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"tasks.max": "1",
"database.hostname": "mysql",
"database.port": "3306",
"database.user": "foo",
"database.password": "bar",
"database.server.id": "12345",
"database.allowPublicKeyRetrieval":"true",
"database.server.name": "brokername",
"database.whitelist": "testdb",
"table.whitelist": "testdb.planes",
"database.include.list": "testdb",
"snapshot.mode": "schema_only",
"database.history.kafka.bootstrap.servers": <broker>,
"database.history.kafka.topic": "schema-changes.testdb",
"database.history.consumer.security.protocol": "SASL_SSL",
"database.history.consumer.ssl.endpoint.identification.algorithm": "https",
"database.history.consumer.sasl.mechanism": "SCRAM-SHA-256",
"database.history.consumer.sasl.jaas.config": <config>
"database.history.producer.security.protocol": "SASL_SSL",
"database.history.producer.ssl.endpoint.identification.algorithm": "https",
"database.history.producer.sasl.mechanism": "SCRAM-SHA-256",
"database.history.producer.sasl.jaas.config": <config>
}
}
I start up MySQL and create the database and set the permissions
CREATE DATABASE testdb;
GRANT ALL ON *.* TO 'foo'@'%';
FLUSH PRIVILEGES;
Then I start the connector and apply the configuration to it. At that point in time I see throughput in the connect-configs
and connect-statuses
topics.
Next, I create a table.
use testdb;
CREATE TABLE planes (color INT);
As expected, I get messages in the schema-changes.testdb
-topic.
But, when I insert a value with
INSERT INTO planes (color) VALUES (2452);
I would have expected messages in topic brokername.testdb.planes
, but I am not getting anything there.
The connector is constantly repeating the following output
connect | 2022-04-21 05:55:03,664 INFO || [Producer clientId=connector-producer-tutorial-connector-0] Cancelled in-flight API_VERSIONS request with correlation id 7371 due to node -1 being disconnected (elapsed time since creation: 45ms, elapsed time since send: 45ms, request timeout: 30000ms) [org.apache.kafka.clients.NetworkClient]
I am running out of ideas of where or what to troubleshoot. Any pointers that could help me troubleshoot the problem are highly welcome.
Let me know if more info is needed.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|