python - Azure Python 下载存储 blob 返回 'The condition specified using HTTP conditional header(s) is not met.'

Question

使用以下代码:

import os
from azure.identity import (
    ClientSecretCredential
)

# Import the client object from the Azure library
from azure.storage.blob import BlobClient

t_id = "<tenant_id>"
c_id = "<client_id>"
s_acct = "<storage_account_name>"
s_acct_url = "%s.blob.core.windows.net" % s_acct
sek = "<client_sekret>"

print("+ Setup credentials.")
credential = ClientSecretCredential(t_id, c_id, sek)
print("+ Setup Blob Client")
bobc = BlobClient(s_acct_url, <container_name>, <blob_name>,
                  credential=credential)
print("+ Setup streamer")
ssd = bobc.download_blob()
print("+ Get properties")
print(ssd.get_blob_properties())

但我收到以下错误:

$ python azdown.py
+ Setting up stream
+
+ Download stream:
+    Size: 136365212160
Traceback (most recent call last):
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 192, in _download_chunk
    _, response = self.client.download(
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_generated\operations\_blob_operations.py", line 179, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The condition specified using HTTP conditional header(s) is not met.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "azdown.py", line 29, in <module>
    download_stream.download_to_stream(my_blob)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 579, in download_to_stream
    self.readinto(stream)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 561, in readinto
    downloader.process_chunk(chunk)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 125, in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 201, in _download_chunk
    process_storage_error(error)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_shared\response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ResourceModifiedError: The condition specified using HTTP conditional header(s) is not met.
RequestId:<request id>
Time:2021-01-10T01:23:24.8981731Z
ErrorCode:ConditionNotMet
Error:None

我查看了 [1]，但我正在使用 download_to_stream()。然后我尝试了 [2]，但我的存储是第 1 代容器。有一个链接表明它与容器的网络权限有关，但容器设置为允许所有下载。问题是，它确实下载到某个点(atm，120G 中的 1.2G)，然后就卡住了。所以这不是权限问题，而是某种竞争条件。

[1] - https://social.msdn.microsoft.com/Forums/azure/en-US/3b4df832-3340-4415-8d93-d71662e1c540/azure-blob-the-condition-specified-using-http-conditional-headers-is-not-met

[2] - https://forums.databricks.com/questions/20415/intermittent-http-error-when-loading-files-from-ad.html

[编辑]

我还尝试使用connection_string方法:

credential = DefaultAzureCredential()

    blob_url = "DefaultEndpointsProtocol=https;AccountName=<storage_acct>;" + \
               "AccountKey=<account_key>;" + \
               "EndpointSuffix=core.windows.net"

    # Create the client object using the storage URL and the credential
    blob_client = BlobClient.from_connection_string(
        blob_url,
        container_name=<container_name>,
        blob_name=<blob_item>)

    if download_now:
        with open(<blob_item>, "wb") as my_blob:
            print("+ Setting up stream")
            download_stream = blob_client.download_blob()
            print("+")
            print("+ Download stream:")
            print("+    Size: %d" % download_stream.size)
            download_stream.download_to_stream(my_blob)

    else:
        print(blob_client.get_blob_properties())

即使在直播了大约一场演出后，我也会遇到同样的错误。

错误:

Traceback (most recent call last):
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 192, in _download_chunk
    _, response = self.client.download(
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_generated\operations\_blob_operations.py", line 179, in download
    raise models.StorageErrorException(response, self._deserialize)
azure.storage.blob._generated.models._models_py3.StorageErrorException: Operation returned an invalid status 'The condition specified using HTTP conditional header(s) is not met.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "azuredl.py", line 52, in <module>
    download_stream.download_to_stream(my_blob)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 579, in download_to_stream
    self.readinto(stream)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 561, in readinto
    downloader.process_chunk(chunk)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 125, in process_chunk
    chunk_data = self._download_chunk(chunk_start, chunk_end - 1)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_download.py", line 201, in _download_chunk
    process_storage_error(error)
  File "C:\git\python38\lib\site-packages\azure\storage\blob\_shared\response_handlers.py", line 147, in process_storage_error
    raise error
azure.core.exceptions.ResourceModifiedError: The condition specified using HTTP conditional header(s) is not met.
RequestId:<request id>
Time:2021-01-10T07:43:24.4927426Z
ErrorCode:ConditionNotMet
Error:None

有人有什么想法吗？

Answer 1

当我对正在下载的文件进行一些更新(例如更改内容或添加一些元数据)时，我可以重现此问题:

此问题是由于 Blob 更新后，其 Etag 同时发生变化。这导致了这个问题。详情见this doc .

要解决此问题，您可以要求 Lise为这个blob添加一个锁，这样这个blob在下载时就不会被编辑或删除。

尝试下面的代码:

blob_client = BlobClient.from_connection_string(conn_str='', container_name='',blob_name='')

#require lease that never expires
lease = blob_client.acquire_lease(lease_duration=-1)

with open("<some path>", "wb") as my_blob:
            print("+ Setting up stream")
            download_stream = blob_client.download_blob()
            print("+")
            print("+ Download stream:")
            print("+    Size: %d" % download_stream.size)
            download_stream.download_to_stream(my_blob)

#break lease after download
lease.break_lease()