python - 无法将较大的 blob 上传到 Azure : azure. core.exceptions.ServiceRequestError : The operation did not complete (write) (_ssl. c:2317)

Question

我正在尝试使用 Python SDK 将一些较大的 blob (>50MB) 上传到我的 Azure 存储容器:

connect_str = os.environ['AZURE_STORAGE_CONNECTION_STRING']
blob_service_client = BlobServiceClient.from_connection_string(connect_str)

def upload_blob(file_path):
    if os.path.exists(file_path):
        with open(file_path, 'rb') as data:
            blob_client = blob_service_client.get_blob_client(container='foo', blob=file_path)

            print(f"Uploading file {file_path} to blob storage...")
            print(os.path.getsize(file_path))
            return blob_client.upload_blob(data, length=os.path.getsize(file_path))
    else:
        print(f"File {file_path} not found. Please store the file first before uploading")
        return False

但是，当我运行此命令时，我收到一个azure.core.exceptions.ServiceRequestError:

Traceback (most recent call last):
  File "C:/Users/.../storage_controller.py", line 96, in <module>
    upload_blob(config.VECTORIZER_PATH)
  File "C:/Users/.../storage_controller.py", line 34, in upload_blob
    return blob_client.upload_blob(data, length=os.path.getsize(file_path))
  File "C:\Users\...\venv\lib\site-packages\azure\core\tracing\decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_blob_client.py", line 496, in upload_blob
    return upload_block_blob(**options)
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_upload_helpers.py", line 104, in upload_block_blob
    **kwargs)
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_generated\operations\_block_blob_operations.py", line 207, in upload
    pipeline_response = self._client._pipeline.run(request, stream=False, **kwargs)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 211, in run
    return first_node.send(pipeline_request)  # type: ignore
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  [Previous line repeated 4 more times]
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\policies\_redirect.py", line 157, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 515, in send
    raise err
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 489, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_shared\policies.py", line 290, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 71, in send
    response = self.next.send(request)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\_base.py", line 103, in send
    self._sender.send(request.http_request, **request.context.options),
  File "C:\Users\...\venv\lib\site-packages\azure\storage\blob\_shared\base_client.py", line 312, in send
    return self._transport.send(request, **kwargs)
  File "C:\Users\...\venv\lib\site-packages\azure\core\pipeline\transport\_requests_basic.py", line 284, in send
    raise error
azure.core.exceptions.ServiceRequestError: The operation did not complete (write) (_ssl.c:2317)

我尝试了一些方法，发现了一些关于分块和使用 put_blob 方法处理较大文件的建议，但这些解决方案在当前版本的 SDK 中似乎无法实现它应该自己处理较大的文件。不过，较小的文件(例如一行的 .txt 文件)绝对可以正常工作。这是 Azure SDK 的问题还是我自己的网络/SSL 配置错误？我该如何解决这个问题？

提前致谢!

Answer 1

我总结了解决方案如下。

如果您想使用azure.storage.blob包将文件以 block 的形式上传到Azure blob，我们可以使用BlobClient.stage_block方法来上传每个 block 。上传后，我们使用 BlobClient.commit_block_list 方法将所有 block 组成一个 blob。

例如

# Instantiate a new BlobServiceClient using a connection string
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
# Instantiate a new ContainerClient
container_client = blob_service_client.get_container_client('')
blob_client = container_client.get_blob_client("csvfile.csv")
# upload data
block_list=[]
chunk_size=1024
with open('csvfile.csv','rb') as f:
   
   while True:
        read_data = f.read(chunk_size)
        if not read_data:
            break # done
        blk_id = str(uuid.uuid4())
        blob_client.stage_block(block_id=blk_id,data=read_data) 
        block_list.append(BlobBlock(block_id=blk_id))
        

blob_client.commit_block_list(block_list)

更多详情请引用here