java - Google PubSub 和来自 TOPIC 的重复消息

Question

如何防止在 Google Cloud PubSub 中出现重复的消息？

比如说，我有一个代码可以处理它订阅的消息。

比如，我有 2 个节点具有具有此代码的相同服务。

一旦一个节点收到消息但尚未确认，另一个节点将收到相同的消息。这就是问题所在，我们有两个重复的消息。

void messageReceiver(PubsubMessage pubsubMessage, AckReplyConsumer ackReply) {

        submitHandler.handle(toMessage(pubsubMessage))
                .doOnSuccess((response) -> {
                    log.info("Acknowledging the successfully processed message id: {}, response {}", pubsubMessage.getMessageId(), response);
                    ackReply.ack();  // <---- acknowledged
                })
                .doOnError((e) -> {
                    log.error("Not acknowledging due to an exception", e);
                    ackReply.nack();
                })
                .doOnTerminate(span::finish)
                .subscribe();
    }

解决这个问题的方法是什么？这是正常行为吗？

Answer 1

Google Cloud Pub/Sub 使用“至少一次”交付。来自 the docs :

Typically, Cloud Pub/Sub delivers each message once and in the order in which it was published. However, messages may sometimes be delivered out of order or more than once. In general, accommodating more-than-once delivery requires your subscriber to be idempotent when processing messages.

这意味着它保证它会传递消息 1:N 次，因此如果您不通过其他首先删除重复数据的东西来传输消息，您可能会多次收到消息。没有您可以定义的设置来保证一次交付。文档确实提到您可以使用 Cloud Dataflow 的 PubSubIO 获得您想要的行为，但是 that solution appears to be deprecated :

You can achieve exactly once processing of Cloud Pub/Sub message streams using Cloud Dataflow PubsubIO. PubsubIO de-duplicates messages on custom message identifiers or those assigned by Cloud Pub/Sub.

综上所述，我实际上从未见过 Google Cloud Pub/Sub 发送消息两次。您确定这真的是您遇到的问题，还是因为您没有在确认截止日期(如上所述，默认为 10 秒)内确认该消息而重新发出消息。如果您不承认，它将被重新发行。来自 the docs (强调我的):

A subscription is created for a single topic. It has several properties that can be set at creation time or updated later, including:

• An acknowledgment deadline: If your code doesn't acknowledge the message before the deadline, the message is sent again. The default is 10 seconds. The maximum custom deadline you can specify is 600 seconds (10 minutes).

如果是这种情况，只需在截止日期前确认您的消息，您就不会经常看到这些重复消息。