sql-server - SQL Server的Debezium问题

Question

我使用了debezium.io here提供的教程，并对其进行了调整，使其使用MS SQL Server数据库而不是MySQL，但是当我对数据库进行更改时，观察程序不会显示任何事件或 Activity 。这是我采取的步骤:

我运行了zookeeper docker命令:
docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1

然后我运行了kafka docker命令:
docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.1

然后我运行了启用了代理的SQL Server docker命令:
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -e 'MSSQL_AGENT_ENABLED=True' --name mssql -p 1433:1433 -d mcr.microsoft.com/mssql/server:2019-latest

，然后我连接到SQL Server实例，并通过运行以下查询创建了一个名为 PeopleDb 的数据库，并创建了一个名为 People 的表:
USE [PeopleDb] GOCREATE TABLE [dbo].[People]([Id] [bigint] IDENTITY(1,1) NOT NULL,[FirstName] [varchar](50) NOT NULL, [LastName] [varchar](50) NOT NULL, CONSTRAINT [PK_People] PRIMARY KEY CLUSTERED ( [Id] ASC)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]) ON [PRIMARY] GO

，然后运行以下命令启用CDC:
Use PeopleDb GoEXEC sys.sp_cdc_enable_dbEXEC sys.sp_cdc_enable_table @source_schema = N'dbo', @source_name = N'People', @role_name = Null, @filegroup_name = N'Primary',@supports_net_changes = 0EXEC sys.sp_cdc_help_change_data_capture

我注意到cdc表是在“系统表”下创建的。当我运行以下查询时，一条记录已添加到 cdc.dbo_People_CT 表中:
INSERT INTO TABLE dbo.People(FirstName, LastName) values ('John', 'Smith')

然后我运行了连接器docker命令:
docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka --link mssql:mssql debezium/connect:1.1

然后我通过将以下Json张贴到http://localhost:8083/connectors/来部署连接器:

   {
       "name": "people-connector",
       "config": {
           "connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
           "tasks.max": "1",
           "database.hostname": "mssql",
           "database.port": "1433",
           "database.user": "sa",
           "database.password": "yourStrong(!)Password",
           "database.dbname": "PeopleDb",
           "database.server.id": "184054",
           "database.server.name": "mssql",
           "database.history.kafka.bootstrap.servers": "kafka:9092",
           "database.history.kafka.topic": "mssql.dbo.people",
           "name": "people-connector"
       },
       "tasks": [],
       "type": "source"
   }

我通过检查http://localhost:8083/connectors/people-connector/status验证了新添加的连接器正在运行:
{"name":"people-connector","connector":{"state":"RUNNING","worker_id":"172.17.0.5:8083"},"tasks":[{"id":0,"state":"RUNNING","worker_id":"172.17.0.5:8083"}],"type":"source"}

我运行了watcher docker命令:
docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1 watch-topic -a -k mssql.dbo.people

产生以下内容:

WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
Using ZOOKEEPER_CONNECT=172.17.0.2:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.6:9092
Using KAFKA_BROKER=172.17.0.3:9092
Contents of topic mssql.dbo.people:
null    {
  "source" : {
    "server" : "mssql"
  },
  "position" : {
    "transaction_id" : null,
    "event_serial_no" : 1,
    "commit_lsn" : "00000025:000003f8:0003",
    "change_lsn" : "NULL"
  },
  "databaseName" : "PeopleDb",
  "schemaName" : "dbo",
  "tableChanges" : [ {
    "type" : "CREATE",
    "id" : "\"PeopleDb\".\"dbo\".\"People\"",
    "table" : {
      "defaultCharsetName" : null,
      "primaryKeyColumnNames" : [ "Id" ],
      "columns" : [ {
        "name" : "Id",
        "jdbcType" : -5,
        "typeName" : "bigint identity",
        "typeExpression" : "bigint identity",
        "charsetName" : null,
        "length" : 19,
        "scale" : 0,
        "position" : 1,
        "optional" : false,
        "autoIncremented" : false,
        "generated" : false
      }, {
        "name" : "FirstName",
        "jdbcType" : 12,
        "typeName" : "varchar",
        "typeExpression" : "varchar",
        "charsetName" : null,
        "length" : 50,
        "position" : 2,
        "optional" : false,
        "autoIncremented" : false,
        "generated" : false
      }, {
        "name" : "LastName",
        "jdbcType" : 12,
        "typeName" : "varchar",
        "typeExpression" : "varchar",
        "charsetName" : null,
        "length" : 50,
        "position" : 3,
        "optional" : false,
        "autoIncremented" : false,
        "generated" : false
      } ]
    }
  } ]
}

完成这些步骤并确保我的连接器正在运行之后，我希望在将新记录插入到People表中或删除或更新它们时看到新事件，但观察者未显示任何 Activity 。有谁知道为什么Debezium和SQL Server之间似乎断开连接？

Answer 1

根据Debezium SQL Server Connector docs:

The SQL Server connector writes events for all insert, update, and delete operations on a single table to a single Kafka topic. The name of the Kafka topics always takes the form serverName.schemaName.tableName, where serverName is the logical name of the connector as specified with the database.server.name configuration property, schemaName is the name of the schema where the operation occurred, and tableName is the name of the database table on which the operation occurred.

在您的情况下，要在 dbo.People数据库中查看 mssql表的更改事件，您需要观看 mssql.dbo.People(请注意，Kafka中的主题名称区分大小写)。