azure - 如何解决Azure函数 `Stream Too Long`错误

Question

我目前正在致力于将数据从 Azure Blob 存储同步到 GCP 存储桶。我们希望在填充 blob 后立即执行此同步操作。

我已决定使用 BlobTriggered Azure Function，它可以按预期工作，但一旦文件相当大(例如 1.5GB？)，事情就会开始失败。最初，我认为这是因为我的代码没有优化。

经过各种排查，我发现即使单独监听blob事件，该功能也失败。您将在下面看到我没有对 blob 进行任何处理，但它失败了。

def main(myblob: func.InputStream):
      logging.info(f"Python blob trigger function processed blob \n"
                 f"Name: {myblob.name}\n"
                 f"Blob Size: {myblob.length} bytes")

        )

我所做的只是记录 blob 的名称和大小，但出现以下错误。

2021-11-11 15:49:38.937
Executing 'Functions.BlobTrigger1' (Reason='New blob detected: tokboxlog/47189174/test-dir-three-ssalami/arrow-larg-two.avi', Id=2dea7d82-eaa3-484f-a47a-c386aca5d926)
Information
2021-11-11 15:49:38.941
Trigger Details: MessageId: 39e440c1-a5ef-4ff4-b2ba-010fa2373759, DequeueCount: 2, InsertionTime: 2021-11-11T15:37:06.000+00:00, BlobCreated: 2021-11-09T13:27:44.000+00:00, BlobLastModified: 2021-11-09T13:27:44.000+00:00
Information
2021-11-11 15:49:53.434
python exited with code 137
Error
2021-11-11 15:49:53.436
Executed 'Functions.BlobTrigger1' (Failed, Id=2dea7d82-eaa3-484f-a47a-c386aca5d926, Duration=134084ms)
Error
2021-11-11 15:49:54.121
python exited with code 137

2021-11-11 15:46:29.918
Executing 'Functions.BlobTrigger1' (Reason='New blob detected: Stream was too long.', Id=3af25993-ffb5-43b7-83ed-ae5d0448757f)
Information
2021-11-11 15:46:29.919
Trigger Details: MessageId: a7c095ba-5879-4a57-a267-8a5debfd935f, DequeueCount: 5, InsertionTime: 2021-11-11T15:37:06.000+00:00, BlobCreated: 2021-11-09T13:26:30.000+00:00, BlobLastModified: 2021-11-09T13:26:30.000+00:00
Information
2021-11-11 15:46:30.159
Stream was too long.
Error
2021-11-11 15:46:30.160
Executed 'Functions.BlobTrigger1' (Failed, Id=3af25993-ffb5-43b7-83ed-ae5d0448757f, Duration=91987ms)
Error
2021-11-11 15:46:30.161
Stream was too long.

无论 SKU 如何，这都会失败。 EP3、P2V2 失败。

Answer 1

它不适用于大型文件，因为您的函数将接收整个流并尝试在内存中分配它。您需要读取函数代码中的 block 内容，或者执行前面的步骤将大文件拆分为较小的文件