gpt4 book ai didi

Celery 任务中每个查询的 Python cassandra-driver OperationTimeOut

转载 作者:太空狗 更新时间:2023-10-29 21:34:27 24 4
gpt4 key购买 nike

我在 celery 任务中异步执行的每个插入查询(小查询)都有问题。在同步模式下,当我插入时一切都很好,但是当它在 apply_async() 中执行时,我得到这个:

OperationTimedOut('errors=errors=errors={}, last_host=***.***.*.***, last_host=None, last_host=None',)

回溯:

Traceback (most recent call last):
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
return self.run(*args, **kwargs)
File "/var/nfs_www/***/www_v1/app/mods/news_feed/tasks.py", line 26, in send_new_comment_reply_notifications
send_new_comment_reply_notifications_method(comment_id)
File "/var/nfs_www/***www_v1/app/mods/news_feed/methods.py", line 83, in send_new_comment_reply_notifications
comment_type='comment_reply'
File "/var/nfs_www/***/www_v1/app/mods/news_feed/models/storage.py", line 129, in add
CommentsFeed(**kwargs).save()
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/models.py", line 531, in save
consistency=self.__consistency__).save()
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/query.py", line 907, in save
self._execute(insert)
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/query.py", line 786, in _execute
tmp = execute(q, consistency_level=self._consistency)
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cqlengine/connection.py", line 95, in execute
result = session.execute(query, params)
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cassandra/cluster.py", line 1103, in execute
result = future.result(timeout)
File "/var/nfs_www/***/env_v0/local/lib/python2.7/site-packages/cassandra/cluster.py", line 2475, in result
raise OperationTimedOut(errors=self._errors, last_host=self._current_host)
OperationTimedOut: errors={}, last_host=***.***.*.***

有人对问题有想法吗?

我找到了这个 When cassandra-driver was executing the query, cassandra-driver returned error OperationTimedOut ,但我的查询很少,而且只在 celery 任务中出现问题。

更新:

我做了一个测试任务,它也引发了这个错误。

@celery.task()
def test_task_with_cassandra():
from app import cassandra_session
cassandra_session.execute('use news_feed')
return 'Done'

更新 2:做了这个:

@celery.task()
def test_task_with_cassandra():
from cqlengine import connection
connection.setup(app.config['CASSANDRA_SERVERS'], port=app.config['CASSANDRA_PORT'],
default_keyspace='test_keyspace')
from .models import Feed
Feed.objects.count()
return 'Done'

明白了:

NoHostAvailable('Unable to connect to any servers', {'***.***.*.***': OperationTimedOut('errors=errors=Timed out creating connection, last_host=None, last_host=None',)})

从 shell 我可以连接到它

更新 3:从 github 问题上删除的线程(在我的电子邮件中找到这个):(这对我也有用)下面是我如何将 CQLengine 插入 Celery:

from celery import Celery
from celery.signals import worker_process_init, beat_init
from cqlengine import connection
from cqlengine.connection import (
cluster as cql_cluster, session as cql_session)

def cassandra_init():
""" Initialize a clean Cassandra connection. """
if cql_cluster is not None:
cql_cluster.shutdown()
if cql_session is not None:
cql_session.shutdown()
connection.setup()

# Initialize worker context for both standard and periodic tasks.
worker_process_init.connect(cassandra_init)
beat_init.connect(cassandra_init)

app = Celery()

This is crude, but works. Should we add this snippet in the FAQ ?

最佳答案

我遇到了类似的问题。这似乎与在任务之间共享 Cassandra session 有关。我通过为每个线程创建一个 session 来解决它。确保从任务中调用 get_session() 然后执行此操作:

thread_local = threading.local()

def get_session():
if hasattr(thread_local, "cassandra_session"):
return thread_local.cassandra_session

cluster = Cluster(settings.CASSANDRA_HOSTS)
session = cluster.connect(settings.CASSANDRA_KEYSPACE)

thread_local.cassandra_session = session

return session

关于Celery 任务中每个查询的 Python cassandra-driver OperationTimeOut,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24785299/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com