python - BashOperator 为其他 PythonOperators 中使用的库引发 ImportError

Question

我的 dag 构建器模块中有一组任务，该模块使用 Airflow 中全局使用的 Python 运算符。我正在 kubernetes 上使用 docker 部署 Airflow 。

任务失败并显示错误消息:没有名为 pandas 的模块。使用 pandas 的其他任务均成功。

是的，我确实进入了容器(workers)并发现pip3 freeze确实显示了pandas。

2018-12-13 12:30:23,332] {bash_operator.py:87} INFO - Temporary script location: /tmp/airflowtmppkovwfth/pscript_pclean_zjg4qfamp9pda9jsxysyrqfj_AWFtK5ucowyw2
[2018-12-13 12:30:23,333] {bash_operator.py:97} INFO - Running command: python /usr/local/airflow/rootfs/mopng_baseline_v2/scripts/pclean_zjg4qfamp9pda9jsxysyrqfj_AWFtK.py /usr/local/airflow/rootfs/mopng_baseline_v2/scheduled__2018-12-12T14:00:00+00:00/appended/DsDnV0TjSHnL0DF53JLjmUtO.csv /usr/local/airflow/rootfs/mopng_baseline_v2/scheduled__2018-12-12T14:00:00+00:00/pcleaned/ztYVV9nkh5t425gYjFqKuAD9.csv
[2018-12-13 12:30:23,344] {bash_operator.py:106} INFO - Output:
[2018-12-13 12:30:23,359] {bash_operator.py:110} INFO - Traceback (most recent call last):
[2018-12-13 12:30:23,359] {bash_operator.py:110} INFO -   File "/usr/local/airflow/rootfs/mopng_baseline_v2/scripts/pclean_zjg4qfamp9pda9jsxysyrqfj_AWFtK.py", line 3, in <module>
[2018-12-13 12:30:23,359] {bash_operator.py:110} INFO -     import pandas as pd
[2018-12-13 12:30:23,360] {bash_operator.py:110} INFO - ImportError: No module named pandas
[2018-12-13 12:30:23,362] {bash_operator.py:114} INFO - Command exited with return code 1
[2018-12-13 12:30:23,383] {models.py:1736} ERROR - Bash command failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1633, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.5/dist-packages/airflow/operators/bash_operator.py", line 118, in execute
    raise AirflowException("Bash command failed")
airflow.exceptions.AirflowException: Bash command failed

Answer 1

失败的操作符不是 PythonOperator，而是 BashOperator。最可能的原因是 Bash 中的 python 当前指向与运行 Airflow 的 Python 环境不同的 Python 环境。

请务必在 BashOperator 中指定 python3，或者在与 PythonOperator< 相同的环境中从命令行调用 Python 所需的任何额外配置 确实如此。