docker - Airflow :DockerOperator 失败并显示权限被拒绝错误

Question

我正在尝试通过 Airflow 运行 docker 容器，但得到 Permission Denied错误。我看过一些相关的帖子，有些人似乎已经通过sudo chmod 777 /var/run/docker.sock 解决了它这充其量是一个有问题的解决方案，但它仍然对我不起作用(即使在重新启动 docker 之后。如果有人设法解决了这个问题，请告诉我!
这是我的 DAG:

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.docker_operator import DockerOperator

args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2020, 6, 21, 11, 45, 0),
    'retries': 1,
    'retry_delay': timedelta(minutes=1),
}

dag = DAG(
    "docker",
    default_args=args,
    max_active_runs=1,
    schedule_interval='* * * * *',
    catchup=False
)

hello_operator = DockerOperator(
    task_id="run_docker",
    image="alpine:latest",
    command="/bin/bash echo HI!",
    auto_remove=True,
    dag=dag
)

这是我得到的错误:

[2020-06-21 14:01:36,620] {taskinstance.py:1145} ERROR - ('Connection aborted.', PermissionError(13, 'Permission denied'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
PermissionError: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/airflow/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.6/http/client.py", line 1262, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1308, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1257, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 974, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.6/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', PermissionError(13, 'Permission denied'))

这是我的设置:
Dockerfile:

FROM apache/airflow
RUN pip install --upgrade --user pip && \
    pip install --user psycopg2-binary && \
    pip install --user docker
COPY airflow/airflow.cfg /opt/airflow/

docker-compose.yaml:

version: "3"

services:

  postgres:
    image: "postgres:9.6"
    container_name: "postgres"
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    ports:
    - "5432:5432"
    volumes:
    - ./data/postgres:/var/lib/postgresql/data

  initdb:
    image: learning/airflow
    entrypoint: airflow initdb
    depends_on:
      - postgres

  webserver:
    image: learning/airflow
    restart: always
    entrypoint: airflow webserver
    environment:
      - EXECUTOR=Local
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
      interval: 1m
      timeout: 5m
      retries: 3
    ports:
      - "8080:8080"
    depends_on:
      - postgres
    volumes:
    - ./airflow/dags:/opt/airflow/dags
    - ./airflow/plugins:/opt/airflow/plugins
    - ./data/logs:/opt/airflow/logs
    - /var/run/docker.sock:/var/run/docker.sock

  scheduler:
    image: learning/airflow
    restart: always
    entrypoint: airflow scheduler
    healthcheck:
      test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-scheduler.pid ]"]
      interval: 1m
      timeout: 5m
      retries: 3
    depends_on:
      - postgres
    volumes:
      - ./airflow/dags:/opt/airflow/dags
      - ./airflow/plugins:/opt/airflow/plugins
      - ./data/logs:/opt/airflow/logs
      - /var/run/docker.sock:/var/run/docker.sock

Answer 1

您可以尝试使用以下命令运行 docker 文件:

docker run -v /var/run/docker.sock:/var/run/docker.sock your_image_name