'Debezium MySQL not capturing INSERT operations

I am running MySQL in a single instance and Debezium as Docker-containers on my local machine via docker-compose

mysql:
    image: mysql:latest
    container_name: mysql
    command:
      --server-id=12345
      --log-bin=mysql-bin
      --binlog-format=ROW
      --binlog-row-image=full
    networks:
      - tutorial
    environment:
      MYSQL_USER: foo
      MYSQL_PASSWORD: bar
      MYSQL_ROOT_PASSWORD: foobar
    volumes:
      - db_data:/var/lib/mysql


connect:
    image: quay.io/debezium/connect:1.9
    container_name: connect
    ports:
      - "8083:8083"
    networks:
      - tutorial
    environment:
      BOOTSTRAP_SERVERS: <broker>
      
      CONFIG_STORAGE_TOPIC: connect-configs
      OFFSET_STORAGE_TOPIC: connect-offsets
      STATUS_STORAGE_TOPIC: connect-statuses

      CONNECT_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_SASL_MECHANISM: SCRAM-SHA-256
      CONNECT_SASL_JAAS_CONFIG: <config>

volumes:
  db_data:
  
networks:
  tutorial:

The Kafka broker is on Upstash where I have created a cluster with the following topics:

  • connect-offsets
  • connect-statuses
  • connect-configs
  • schema-changes.testdb
  • brokername.testdb.planes

My connector config looks as follows

{
  "name": "tutorial-connector",  
  "config": {  
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "tasks.max": "1",  
    "database.hostname": "mysql",  
    "database.port": "3306",
    "database.user": "foo",
    "database.password": "bar",
    "database.server.id": "12345",
    "database.allowPublicKeyRetrieval":"true",
    "database.server.name": "brokername",
    "database.whitelist": "testdb",
    "table.whitelist": "testdb.planes",
    "database.include.list": "testdb", 
    "snapshot.mode": "schema_only",
    "database.history.kafka.bootstrap.servers": <broker>,  
    "database.history.kafka.topic": "schema-changes.testdb",
    "database.history.consumer.security.protocol": "SASL_SSL",
    "database.history.consumer.ssl.endpoint.identification.algorithm": "https",
    "database.history.consumer.sasl.mechanism": "SCRAM-SHA-256",
    "database.history.consumer.sasl.jaas.config": <config>
    "database.history.producer.security.protocol": "SASL_SSL",
    "database.history.producer.ssl.endpoint.identification.algorithm": "https",
    "database.history.producer.sasl.mechanism": "SCRAM-SHA-256",
    "database.history.producer.sasl.jaas.config": <config>
  }
}

I start up MySQL and create the database and set the permissions

CREATE DATABASE testdb;
GRANT ALL ON *.* TO 'foo'@'%';
FLUSH PRIVILEGES;

Then I start the connector and apply the configuration to it. At that point in time I see throughput in the connect-configs and connect-statuses topics.

Next, I create a table.

use testdb;
CREATE TABLE planes (color INT);

As expected, I get messages in the schema-changes.testdb-topic.

But, when I insert a value with

INSERT INTO planes (color) VALUES (2452);

I would have expected messages in topic brokername.testdb.planes, but I am not getting anything there.

The connector is constantly repeating the following output

connect      | 2022-04-21 05:55:03,664 INFO   ||  [Producer clientId=connector-producer-tutorial-connector-0] Cancelled in-flight API_VERSIONS request with correlation id 7371 due to node -1 being disconnected (elapsed time since creation: 45ms, elapsed time since send: 45ms, request timeout: 30000ms)   [org.apache.kafka.clients.NetworkClient]

I am running out of ideas of where or what to troubleshoot. Any pointers that could help me troubleshoot the problem are highly welcome.

Let me know if more info is needed.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source