python - Dask:如何向 Dask 集群添加安全性 (TLS/SSL)？

Question

我试图弄清楚如何在 GCP 上使用 GKE 上的 helm 部署的 Dask 集群中添加安全层，这将强制用户将证书和 key 文件输入到安全对象中，如本文档 [1] 中所述.不幸的是，调度程序 pod 崩溃时出现超时错误。调查日志后，错误如下:

Traceback (most recent call last):
  File "/opt/conda/bin/dask-scheduler", line 10, in <module>
    sys.exit(go())
  File "/opt/conda/lib/python3.7/site-packages/distributed/cli/dask_scheduler.py", line 226, in go
    main()
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/distributed/cli/dask_scheduler.py", line 206, in main
    **kwargs
  File "/opt/conda/lib/python3.7/site-packages/distributed/scheduler.py", line 1143, in __init__
    self.connection_args = self.security.get_connection_args("scheduler")
  File "/opt/conda/lib/python3.7/site-packages/distributed/security.py", line 224, in get_connection_args
    "ssl_context": self._get_tls_context(tls, ssl.Purpose.SERVER_AUTH),
  File "/opt/conda/lib/python3.7/site-packages/distributed/security.py", line 187, in _get_tls_context
    ctx = ssl.create_default_context(purpose=purpose, cafile=ca)
  File "/opt/conda/lib/python3.7/ssl.py", line 584, in create_default_context
    context.load_verify_locations(cafile, capath, cadata)
FileNotFoundError: [Errno 2] No such file or directory

Helm Config Yaml 文件如下:

scheduler:
  allowed-failures: 5
  env:
    - name: DASK_DISTRIBUTED__COMM__DEFAULT_SCHEME
      value: "tls"
    - name: DASK_DISTRIBUTED__COMM__REQUIRE_ENCRYPTION
      value: "true"
    - name: DASK_DISTRIBUTED__COMM__TLS__CA_FILE
      value: "myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__KEY
      value: "mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__CERT
      value: "myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__KEY
      value: "mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__CERT
      value: "myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__KEY
      value: "mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__CERT
      value: "myca.pem"

我按如下方式创建 key 和证书文件:

openssl req -newkey rsa:4096 -nodes -sha256 -x509 -days 3650 -nodes -out myca.pem -keyout mykey.pem

这是一个最小的完整可验证示例:

import dask.dataframe as dd
from dask.distributed import Client
from distributed.security import Security

sec = Security(tls_ca_file='myca.pem',
               tls_client_cert='myca.pem',
               tls_client_key='mykey.pem',
               require_encryption=True)

with Client("tls://<scheduler_ip>:8786", security=sec) as dask_client:
    ddf = dd.read_csv('gs://<bucket_name>/my_file.csv',
                      engine='python',
                      error_bad_lines=False,
                      encoding="utf-8",
                      assume_missing=True
                      )

    print(ddf.shape[0].compute())

[1] https://distributed.dask.org/en/latest/tls.html

Answer 1

我解决了这个问题。 Dask 工作人员和调度程序都需要在配置中包含证书文件。此外，我们还需要在 dockerfile 中烘焙证书。请参阅下面的完整配置:
Dockerfile

FROM daskdev/dask

RUN conda install --yes \
    -c conda-forge \
    python==3.7

ADD certs /certs/

ENTRYPOINT ["tini", "-g", "--", "/usr/bin/prepare.sh"]

Helm 配置

worker:
  name: worker
  image:
    repository: "gcr.io/PROJECT_ID/mydask"
    tag: "latest"
  env:
    - name: DASK_DISTRIBUTED__COMM__DEFAULT_SCHEME
      value: "tls"
    - name: DASK_DISTRIBUTED__COMM__REQUIRE_ENCRYPTION
      value: "true"
    - name: DASK_DISTRIBUTED__COMM__TLS__CA_FILE
      value: "certs/myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__KEY
      value: "certs/mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__CERT
      value: "certs/myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__KEY
      value: "certs/mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__CERT
      value: "certs/myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__KEY
      value: "certs/mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__CERT
      value: "certs/myca.pem"

scheduler:
  name: scheduler
  image:
    repository: "gcr.io/PROJECT_ID/mydask"
    tag: "latest"
  env:
    - name: DASK_DISTRIBUTED__COMM__DEFAULT_SCHEME
      value: "tls"
    - name: DASK_DISTRIBUTED__COMM__REQUIRE_ENCRYPTION
      value: "true"
    - name: DASK_DISTRIBUTED__COMM__TLS__CA_FILE
      value: "certs/myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__KEY
      value: "certs/mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__SCHEDULER__CERT
      value: "certs/myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__KEY
      value: "certs/mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__WORKER__CERT
      value: "certs/myca.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__KEY
      value: "certs/mykey.pem"
    - name: DASK_DISTRIBUTED__COMM__TLS__CLIENT__CERT
      value: "certs/myca.pem"