java - Kafka 消息版本控制

Question

目前我正在自己实现模式注册表。但也许 kafka 消息(有效负载)版本控制还有其他方法？我了解 confluence.io 架构注册表(以及其他一些类似的注册表)。所以我的问题是:架构注册表是确保消息向后兼容性的唯一可能方法吗？

顺便问一下，有什么好的库可以从对象和模式 json 转换 java 对象吗？

Answer 1

Confluent Schema Registry提供 Apache Kafka 中 Avro 模式的版本控制。

根据documentation ，schema注册表支持4种兼容性配置，默认为向后兼容:

The schema registry server can enforce certain compatibility rules when new schemas are registered in a subject. Currently, we support the following compatibility rules.

Backward compatibility (default): A new schema is backward compatible if it can be used to read the data written in all previous schemas. Backward compatibility is useful for loading data into systems like Hadoop since one can always query data of all versions using the latest schema.

Forward compatibility: A new schema is forward compatible if all previous schemas can read data written in this schema. Forward compatibility is useful for consumer applications that can only deal with data in a particular version that may not always be the latest version.

Full compatibility: A new schema is fully compatible if it’s both backward and forward compatible.

No compatibility: A new schema can be any schema as long as it’s a valid Avro.

您可以将兼容性更改为例如BACKWARD 使用以下调用

# Update compatibility requirements globally
$ curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
    --data '{"compatibility": "BACKWARD"}' \
    http://localhost:8081/config

响应应该是

{"compatibility":"BACKWARD"}

为了将 Java 对象序列化为 Avro，您可以使用 Java Reflection API 。例如，您可以使用以下命令创建架构:

Schema schema = ReflectData.get().getSchema(T);