通过 'eta' 选项进行调度时不遵守 Celeryrate_limit

Question

我想安排一个任务在未来的不同时间运行。我试图在调用任务时使用 eta 选项来执行此操作。这部分本身工作得很好，但同时，我想对任务进行速率限制。我无法同时完成这两件事:安排任务在将来运行，同时也让 celery 遵守速率限制。

这是一个示例(rate_limit_eta.py)

from celery import Celery
from datetime import datetime, timedelta

app = Celery('rate_limit_eta', broker='amqp://guest@localhost//')

@app.task(rate_limit='4/m')
def my_task():
    print 'execution timestamp: {}'.format(datetime.utcnow())


if __name__ == '__main__':

    now = datetime.utcnow()
    d = now + timedelta(seconds=5)
    print now

    for i in range(10):
        my_task.apply_async(eta=d)

在循环中，当我使用 eta=d 调用“my_task”时，任务会尽快执行。不遵守rate_limit。

my_task.apply_async(eta=d)


[2015-06-12 13:45:15,750: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.748498
[2015-06-12 13:45:15,757: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.757757
[2015-06-12 13:45:15,760: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.759992
[2015-06-12 13:45:15,763: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.763614
[2015-06-12 13:45:15,766: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.765996
[2015-06-12 13:45:15,768: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.768424
[2015-06-12 13:45:15,771: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.771079
[2015-06-12 13:45:15,774: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.774561
[2015-06-12 13:45:15,777: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.777425
[2015-06-12 13:45:15,780: WARNING/Worker-1] execution timestamp: 2015-06-12 13:45:15.780207

但是如果我不安排任务(即没有 eta)，则会强制执行速率限制。

my_task.apply_async()

[2015-06-12 13:47:05,844: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.843872
[2015-06-12 13:47:05,850: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:05.850502
[2015-06-12 13:47:21,919: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:21.919120
[2015-06-12 13:47:35,937: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:35.937060
[2015-06-12 13:47:51,959: WARNING/Worker-1] execution timestamp: 2015-06-12 13:47:51.959174
[2015-06-12 13:48:05,976: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:05.976197
[2015-06-12 13:48:21,998: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:21.998254
[2015-06-12 13:48:36,014: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:36.014837
[2015-06-12 13:48:52,041: WARNING/Worker-1] execution timestamp: 2015-06-12 13:48:52.041326
[2015-06-12 13:49:06,057: WARNING/Worker-1] execution timestamp: 2015-06-12 13:49:06.057286

我做错了什么吗？

Answer 1

我也遇到了同样的问题，我找到了原因。我查看了 Celery 代码，这种行为来自 the strategy代码，它执行以下操作:

if req.eta:
    apply_eta()
else:
    check_if_rate_limited()

恐怕 ETA 和速率限制是两种不兼容的方式来告诉 Celery 何时应该执行任务。 Celery 必须选择其中之一，而它恰好是 ETA。

我的链接是 Celery v3(因为我怀疑这个问题是针对 v3 提出的)，但在撰写本文时，这在 Celery 的 master 分支上仍然有效。我想这个限制可以在某个地方更好地记录( here? )，这只是一个拉取请求!

更新:我已经打开an issue关于 Celery 项目。