python - Airflow:如何将 PDF 文件从 S3 存储桶下载到 Airflow

Question

尝试了下面的代码，但收到错误“无法找到凭据”def 下载():

bucket = 'bucketname'
key = 'path and filename'
s3_resource = boto3.resource('s3')
my_bucket = s3_resource.Bucket(bucket)
objects = my_bucket.objects.filter(Prefix=key)
for obj = objects:
    path,filename = os.path.split(obj.key)
    my_bucket.download_file(obj.key, filename)

Answer 1

您需要定义 AWS 连接并使用通过 S3Hook 实现 download_fileobj 函数。

我没有测试它，但它应该是这样的:

from tempfile import NamedTemporaryFile
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook('my_aws_conn')
key_object = hook.get_key('your_path')
with NamedTemporaryFile("wb") as f:
     key_object.download_fileobj(Fileobj=f)
     f.flush()