python - Mrjob 无法在 dataproc : __init__() got an unexpected keyword argument 'channel' 上创建集群

Question

我正在尝试使用 Python mrjob 在 Google Cloud Dataproc 上运行 Hadoop Map Reduce 字数示例图书馆。然而，mrjob失败，但有以下异常:

TypeError: __init__() got an unexpected keyword argument 'channel'
Traceback (most recent call last):
  File "freq.py", line 21, in <module>
    MRWordFreqCount.run()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 616, in run
    cls().execute()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 687, in execute
    self.run_job()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/job.py", line 636, in run_job
    runner.run()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/runner.py", line 503, in run
    self._run()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 468, in _run
    self._launch()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 473, in _launch
    self._launch_cluster()
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 637, in _launch_cluster
    self._get_cluster(self._cluster_id)
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 1188, in _get_cluster
    return self.cluster_client.get_cluster(
  File "/usr/local/lib/python3.8/dist-packages/mrjob/dataproc.py", line 376, in cluster_client
    return google.cloud.dataproc_v1beta2.ClusterControllerClient(
TypeError: __init__() got an unexpected keyword argument 'channel'

我查了一下 GOOGLE_APPLICATION_CREDENTIALS设置正确，在 Google Cloud 上启用了 API，并且为服务帐号设置了所有必需的角色。 mrjob成功将文件上传到 Google Cloud Storage。但是一旦它尝试创建新的 Dataproc 集群就会失败。
什么可能是错误的？
命令行启动 mrjob在 Dataproc 上:

$ python3 freq.py -r dataproc words.txt

当前 Python 环境:

$ python3 -VV
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]

$ pip3 list | grep google
google-api-core          1.23.0
google-auth              1.23.0
google-auth-oauthlib     0.4.2
google-cloud-core        1.4.3
google-cloud-dataproc    2.0.2
google-cloud-logging     1.15.1
google-cloud-storage     1.32.0
google-crc32c            1.0.0
google-pasta             0.2.0
google-resumable-media   1.1.0
googleapis-common-protos 1.52.0

$ pip3 list | grep mrjob
mrjob                    0.7.4

Answer 1

解决方案是将 google-cloud-dataproc 降级到 1.1.1。
在调试到 mrjob 实现后，我发现 mrjob 0.7.4 版使用自 2.0.0 版以来在 google-cloud-dataproc 库上重命名的参数调用 google.cloud.dataproc_v1beta2.ClusterControllerClient 的构造函数。
如何使用 pip3 降级:

$ pip3 install --force-reinstall --no-deps google-cloud-dataproc==1.1.1