python - 强制 celery 使用 StrictRedis

Question

我突然在 celery 中遇到以下错误。

  File "/opt/app-root/lib/python3.6/site-packages/celery/canvas.py", line 636, in __call__                                                              
    return self.apply_async((), {'body': body} if body else {}, **options)                                                                              
  File "/opt/app-root/lib/python3.6/site-packages/celery/canvas.py", line 631, in apply_async                                                           
    parent = _chord(self.tasks, body, args, **options)                                                                                                  
  File "/opt/app-root/lib/python3.6/site-packages/celery/app/task.py", line 420, in __call__                                                            
    return self.run(*args, **kwargs)                                                                                                                    
  File "/opt/app-root/lib/python3.6/site-packages/celery/app/builtins.py", line 348, in run                                                             
    max_retries=max_retries, propagate=propagate, result=results,                                                                                       
  File "/opt/app-root/lib/python3.6/site-packages/celery/backends/base.py", line 537, in _apply_chord_incr                                              
    self.save_group(group_id, self.app.GroupResult(group_id, result))                                                                                   
  File "/opt/app-root/lib/python3.6/site-packages/celery/backends/base.py", line 340, in save_group                                                     
    return self._save_group(group_id, result)                                                                                                           
  File "/opt/app-root/lib/python3.6/site-packages/celery/backends/base.py", line 510, in _save_group                                                    
    self.encode({'result': result.as_tuple()}))                                                                                                         
  File "/opt/app-root/lib/python3.6/site-packages/celery/backends/redis.py", line 161, in set                                                           
    return self.ensure(self._set, (key, value), **retry_policy)                                                                                         
  File "/opt/app-root/lib/python3.6/site-packages/celery/backends/redis.py", line 150, in ensure                                                        
    **retry_policy                                                                                                                                      
  File "/opt/app-root/lib/python3.6/site-packages/kombu/utils/__init__.py", line 246, in retry_over_time                                                
    return fun(*args, **kwargs)                                                                                                                         
  File "/opt/app-root/lib/python3.6/site-packages/celery/backends/redis.py", line 170, in _set                                                          
    pipe.execute()                                                                                                                                      
  File "/opt/app-root/lib/python3.6/site-packages/redis/client.py", line 3443, in execute                                                               
    return execute(conn, stack, raise_on_error)                                                                                                         
  File "/opt/app-root/lib/python3.6/site-packages/redis/client.py", line 3358, in _execute_transaction                                                  
    self.raise_first_error(commands, response)                                                                                                          
  File "/opt/app-root/lib/python3.6/site-packages/redis/client.py", line 3392, in raise_first_error                                                     
    raise r                                                                                                                                             
redis.exceptions.ResponseError: Command # 1 (SETEX b'celery-taskset-meta-20008ab9-37ac-45ae-adce-7085204e0df1' b'\x80\x02}q\x00X\x06\x00\x00\x00resultq\
x01X$\x00\x00\x0020008ab9-37ac-45ae-adce-7085204e0df1q\x02]q\x03(X$\x00\x00\x0041384a3d-b60b-41ce-b39b-10e33815e21eq\x04N\x86q\x05N\x86q\x06X$\x00\x00\x
008182743f-c9fb-48b1-a023-c54dd1431b64q\x07N\x86q\x08N\x86q\tX$\x00\x00\x005fd2949c-2141-4196-8cdd-c16f8a70d1faq\nN\x86q\x0bN\x86q\x0cX$\x00\x00\x00a7aa
bcb6-3784-4a16-b0f4-2ea8b67f2b42q\rN\x86q\x0eN\x86q\x0fX$\x00\x00\x00ee5b8d57-8477-4a84-bb3c-7d52e52dc662q\x10N\x86q\x11N\x86q\x12X$\x00\x00\x00406eda60
-1454-47a9-ba0d-2affa77c62eeq\x13N\x86q\x14N\x86q\x15X$\x00\x00\x00048a5f5f-d2da-4205-99e2-66e91d0f0969q\x16N\x86q\x17N\x86q\x18e\x86q\x19s.' 86400) of 
pipeline caused error: value is not an integer or out of range  

我从其他帖子中了解到，这是因为 strictredis 与 redis。我找不到任何关于如何设置 celery 以使用 strictredis 的文档。

我的celery配置是这样的

import os
from config import debug

if not os.environ.get('REDIS_HOST'):
    REDIS_HOST = '127.0.0.1'
else:
    REDIS_HOST = os.environ.get('REDIS_HOST')
REDIS_PASSWORD = 'foobared' if not os.environ.get('REDIS_PASSWORD') else os.environ.get('REDIS_PASSWORD')

CELERY_ANNOTATIONS = {'tasks.account_lookup': {'rate_limit': '40/m'},
                      'tasks.query': {'rate_limit': '40/m'},
                      'tasks.messages_query': {'rate_limit': '40/m'},
                      'tasks.email': {'rate_limit': '2/s'},
                      'tasks.userpull':{'rate_limit': '1/s'}}
CELERY_TIMEZONE = 'America/Chicago'
CELERY_IMPORTS = ['myapp']
CELERY_BROKER_URL = 'redis://:{0}@{1}:6379/0'.format(REDIS_PASSWORD, REDIS_HOST)
BROKER_URL = 'redis://:{0}@{1}:6379/0'.format(REDIS_PASSWORD, REDIS_HOST)
CELERY_RESULT_BACKEND = 'redis://:{0}@{1}:6379/0'.format(REDIS_PASSWORD, REDIS_HOST)
CELERY_ACCEPT_CONTENT = ['pickle', 'json']
debug = True if os.getenv('CELERY_DEBUG') == 'True' else False
if debug:
    CELERY_ALWAYS_EAGER = True
    CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

我不确定为什么这会随着 redis 服务器发生变化，我的代码也没有发生任何变化。

Answer 1

发现是pip安装的redis版本问题。显然 redis==3.0.0.post1 使用 (key,expire,value) 而 redis==2.10.6 使用 (key,value,expire)。我忘记锁定我的版本，所以当我重新发布我的代码时，redis 模块已更新。