python - Airflow - 外部 API 调用给出 Negsignal.SIGSEGV 错误

Question

我正在使用 Python 脚本调用天气 API，但 Airflow 任务失败并显示错误 Negsignal.SIGSEGV。调用天气 API 的 Python 脚本在 Airflow 外部运行时工作正常。

DAG

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from airflow.contrib.operators.spark_submit_operator import SparkSubmitOperator
from datetime import datetime, timedelta
from scripts.weather_analysis.data_collection import query_weather_data
import pendulum

local_tz = pendulum.timezone("Asia/Calcutta")

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    #'start_date': airflow.utils.dates.days_ago(2), --> doesn't work
    'start_date': datetime(2022, 8, 29, tzinfo=local_tz),
}


dag = DAG('weather_dag_2', default_args=default_args, schedule_interval ='0 * * * *',
    )

# DAG to fetch weather data from api
t1 = PythonOperator(
        task_id = 'callApi',
        python_callable = query_weather_data,
        dag=dag
    )

Python 脚本 - query_weather_data.py

import requests
import json
from scripts.weather_analysis.config import API_KEY
from datetime import datetime

def query_weather_data():

    parameters = {'q':'Brooklyn, USA', 'appId': API_KEY}
    result = requests.get("http://api.openweathermap.org/data/2.5/weather?",parameters)

    if result.status_code == 200:
        json_data = result.json()
        print(json_data)
    else:
        print("Unable to fetch api data")

错误日志:

[2022-09-02, 17:00:04 IST] {local_task_job.py:156} INFO - Task exited with return code Negsignal.SIGSEGV
[2022-09-02, 17:00:04 IST] {taskinstance.py:1407} INFO - Marking task as FAILED. dag_id=weather_dag_2, task_id=callApi, execution_date=20220902T103000, start_date=20220902T113004, end_date=20220902T113004

环境细节:

MacOS 蒙特雷

Airflow =2.3.4

Airflow部署模式=本地

Python=3.10

我已经尝试过此处列出的解决方案 Airflow DAG fails when PythonOperator tries to call API and download data但这并没有解决我的问题。

请帮忙。

Answer 1

我解决了这个解决方法:

在query_weather_data.py中，设置环境变量-

os.environ["no_proxy"]="*"