python - Celery SQS + 任务重复 + SQS 可见性超时

Question

我的大多数 Celery 任务的 ETA 都长于 Amazon SQS 定义的最大可见性超时。

celery documentation说:

This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.

So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use.

同时它还说:

The maximum visibility timeout supported by AWS as of this writing is 12 hours (43200 seconds):

如果我使用 SQS，我应该怎么做才能避免在我的工作人员中多次执行任务？

Answer 1

通常，执行预计到达时间很长的任务不是一个好主意。

首先是“visibility_timeout”问题。而且你可能不想要一个非常大的可见性超时，因为如果工作人员在任务即将运行前 1 分钟崩溃，那么队列仍然会等待 visibility_timeout 完成，然后再将任务发送给另一个工作人员，我猜你不想要这又是 1 个月。

来自 celery 文档:

Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.

而且，SQS 只允许确认列表中的任务数。

SQS 将这些任务称为“飞行消息”。来自 http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html :

A message is considered to be in flight after it's received from a queue by a consumer, but not yet deleted from the queue.

For standard queues, there can be a maximum of 120,000 inflight messages per queue. If you reach this limit, Amazon SQS returns the OverLimit error message. To avoid reaching the limit, you should delete messages from the queue after they're processed. You can also increase the number of queues you use to process your messages.

For FIFO queues, there can be a maximum of 20,000 inflight messages per queue. If you reach this limit, Amazon SQS returns no error messages.

我看到了两种可能的解决方案，您可以改用 RabbitMQ，它不依赖于可见性超时(如果您不想管理自己的服务，可以使用“RabbitMQ 即服务”服务)，或者更改您的代码以具有非常小的 ETA (最佳实践)

这些是我的 2 美分，也许@asksol 可以提供一些额外的见解。