python - 将文件实例作为参数传递给 celery 任务引发 "ValueError: I/O operation on closed file"

Question

我需要将文件作为参数传递给 celery 任务，但传递的文件以某种方式关闭了。以防万一我以异步方式执行任务。这是预期的行为吗？

观看次数:

from engine.tasks import s3_upload_handler
def myfunc():
    f = open('/app/uploads/pic.jpg', 'rb')
    s3_file_handler.apply_async(kwargs={"uploaded_file" : f,"file_name" : "test.jpg"})

任务:

def s3_upload_handler(uploaded_file,file_name):
    ...
    #some code for uploading to s3

回溯:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/photohosting/engine/tasks.py", line 34, in s3_upload_handler
    key.set_contents_from_file(uploaded_file)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1217, in set_contents_from_file
    spos = fp.tell()
ValueError: I/O operation on closed file

花原木:

kwargs  {
         'file_name': 'test.jpg', 
         'uploaded_file': <closed file '<uninitialized file>', 
          mode '<uninitialized file>' at 0x7f6ab9e75e40>
         }

Answer 1

是的，当然，文件会在那里关闭。异步 celery 任务在一个完全独立的进程中运行(此外，它们甚至可以在不同的机器上运行)并且无法将打开的文件传递给它。

您应该在调用任务的过程中关闭文件，然后将其名称和文件中的位置(如果需要)传递给任务，然后在任务中重新打开它。