I'm new to both docker and airflow and I'm trying to create an airflow pipeline in docker on my windows desktop. I have used the latest airflow image and extended it to include some additional python libraries that are needed. All the libraries were installed, image extended and container up and running.
我对docker和airflow都是新手,我正试图在我的Windows桌面上的docker中创建一个气流管道。我使用了最新的Airflow图像,并对其进行了扩展,以包括所需的一些附加的Python库。所有的库都已安装,映像已扩展,容器已启动并运行。
Next, I write the .py file which contains all DAG's and have imported the following modules.
接下来,我编写.py文件,该文件包含所有DAG并导入了以下模块。
from airflow import DAG
from airflow.operators.bash import BashOperator
import time as tm
import pickle, os, datetime, json, requests, locale
from datetime import datetime, date, time, timedelta
from yahoo_fin import options as op
import yfinance as yf
import pandas as pd, numpy as np
from polygon import RESTClient
from polygon.rest.models import Sort
from urllib3 import HTTPResponse
After saving this file, I see the following error message on airflow UI:
保存此文件后,我在AirFlow界面上看到以下错误消息:
Broken DAG: [/opt/airflow/dags/our_first_dag.py] Traceback (most recent call last):
File "", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/our_first_dag.py", line 9, in
from polygon import RESTClient
ImportError: cannot import name 'RESTClient' from 'polygon' (/home/airflow/.local/lib/python3.8/site-packages/polygon/init.py)
中断的DAG:[/opt/airflow/dgs/our_first_dag.py]回溯(最近一次调用):文件“”,第219行,In_Call_With_Frame_Remoted文件“/opt/airflow/dgs/our_first_dag.py”,第9行,从多边形导入RESTClient导入错误:无法从‘POLYGON’(/home/airflow/.local/lib/python3.8/site-packages/polygon/init.py)导入名称‘RESTClient’
From this message it appears that python is unable to import RESTClient from Polygon module, so I check if this package is installed in the image by by going to the container terminal (from docker UI) confirm that all the packages I added while extending the image are actually installed. I also tried running the line which produced error i.e. "from polygon import RESTClient" in the local python and it worked perfectly fine.
从这条消息中可以看出,Python无法从Polygon模块导入RESTClient,所以我通过转到容器终端(从docker UI)确认我在扩展映像时添加的所有包都已实际安装来检查映像中是否安装了此包。我还尝试在本地的python中运行产生错误的代码行,即“from POLYGON IMPORT RESTClient”,它工作得非常好。
I'd appreciate any suggestions on how to fix this.
如果有任何关于如何解决这个问题的建议,我将不胜感激。
Below are the contents of my directory:
以下是我的目录的内容:
dockerfile
扩展坞文件
FROM apache/airflow:2.7.1
COPY requirements.txt /
USER airflow
RUN pip install -r /requirements.txt
USER root
requirements.txt
Requirements.txt
polygon
html5lib
yahoo_fin
yfinance
polygon-api-client
.env
.env
AIRFLOW_UID=50000
docker-compose.yaml
https://pastebin.com/SPNsk1f4
Docker-pose.yaml https://pastebin.com/SPNsk1f4
更多回答
我是一名优秀的程序员,十分优秀!