python - apache beam python 和气流从 GCS 读取结果导致 TypeError ("__init__() got an unexpected keyword argument\' response_encoding\'"

Question

从 Google Cloud Composer 访问 GCS 存储桶会导致类型错误:

DEBUG:apache_beam.io.filesystem:Listing files in 'gs://BUCKET_NAME/FILENAME.2019-01-20'
DEBUG:apache_beam.io.filesystem:translate_pattern: 'gs://BUCKET_NAME/FILENAME.2019-01-20*' -> 'gs\\:\\/\\/BUCKET_NAME\\/FILENAME\\.2019\\-01\\-20[^/\\\\]*'
DEBUG:root:Connecting using Google Application Default Credentials.
File "/usr/local/lib/python2.7/site-packages/apache_beam/io/textio.py", line 524, in __init__
INFO - skip_header_lines=skip_header_lines)
INFO - File "/usr/local/lib/python2.7/site-packages/apache_beam/io/textio.py", line 119, in __init__
INFO - validate=validate)
INFO - File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsource.py", line 121, in __init__
INFO - self._validate()
INFO - File "/usr/local/lib/python2.7/site-packages/apache_beam/options/value_provider.py", line 137, in _f
INFO - return fnc(self, *args, **kwargs)
INFO - File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filebasedsource.py", line 178, in _validate
INFO - match_result = FileSystems.match([pattern], limits=[1])[0]
INFO - File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filesystems.py", line 187, in match
INFO - return filesystem.match(patterns, limits)
INFO - File "/usr/local/lib/python2.7/site-packages/apache_beam/io/filesystem.py", line 705, in match
raise BeamIOError("Match operation failed", exceptions)
apache_beam.io.filesystem.BeamIOError: Match operation failed with exceptions {'gs://BUCKET_NAME/FILENAME.2019-01-20*': BeamIOError('List operation failed with exceptions {\'gs://BUCKET_NAME/FILENAME..2019-01-20\': TypeError("__init__() got an unexpected keyword argument \'response_encoding\'",)}',)}
Command exited with return code 1

该存储桶位于不同的 GCP 项目中，并且 Composer 正在使用具有计算引擎编辑器权限的标准服务帐户运行，该帐户在存储桶中还具有“存储对象查看器”权限。我不知道如何修复此错误，或者它是否是访问权限问题。也许有人可以帮助我或指出我正确的方向？

谢谢

Answer 1

这可以通过向需求添加正确的包来解决。从 Apache Beam 版本 2.9.0 开始，您需要有 apache-beam[gcp] 而不是只有 apache-beam 才能使用 Google Cloud Storage。