amazon-dynamodb - 从 Spark Streaming 中超过 1 个分片的 Kinesis 流中读取

Question

出现以下错误:

com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException:无法更新检查点 - 实例不持有此分片的租约

这可能是什么原因？

Answer 1

这似乎是运动问题。出于调试原因，您可以为 LeaseManager 启用日志记录。有一些额外的消息应该指示检查点请求的状态。需要考虑的重要事项是 failoverTimeMillis。您可以尝试增加 failoverTimeMillis 的值，这有助于避免以下问题。更详细的解释在this GitHub issue .

The root cause of this issue is that sometimes a KCL Worker loses alease and then reacquires a new lease on a particular shard beforediscarding the old lease for that shard, it then continues to hold theold lease which still holds the old concurrency token. The mismatchedconcurrency token prevents checkpointing but the Worker never discardsthe lease because the lease cleanup logic only compares the shard-id.

one way to alleviate the impact of this issue is to increase thefailoverTimeMillis which is set to 1000ms by default in the connectorlibrary. Increasing the failoverTimeMillis will reduce the chance of aWorker losing and then immediately reacquiring its own lease.