airflow - 为什么 Airflow 调度程序一直运行我的 DAG 文件？

Question

我按照教程操作，创建了一个文件夹 $AIRFLOW_HOME/dags，并将教程 DAG python 文件放在那里。然后我启动 Airflow 调度程序。默认情况下它是暂停的。但是如果我查看 Airflow 调度程序的输出，我会看到很多运行，试图创建 DAG。为什么它一直运行？

[2018-09-10 15:49:24,123] {jobs.py:1108} INFO - No tasks to consider for execution.
[2018-09-10 15:49:24,125] {jobs.py:1538} INFO -
================================================================================
DAG File Processing Stats

File Path                                                       PID  Runtime    Last Runtime    Last Run
------------------------------------------------------------  -----  ---------  --------------  -------------------
/Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py  29257  0.44s      0.43s           2018-09-10T13:49:22
================================================================================
[2018-09-10 15:49:24,125] {dag_processing.py:495} INFO - Processor for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py finished
[2018-09-10 15:49:25,133] {dag_processing.py:582} INFO - Started a process (PID: 29258) to generate tasks for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py
[2018-09-10 15:49:25,560] {jobs.py:1108} INFO - No tasks to consider for execution.
[2018-09-10 15:49:25,561] {dag_processing.py:495} INFO - Processor for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py finished
[2018-09-10 15:49:26,567] {dag_processing.py:582} INFO - Started a process (PID: 29259) to generate tasks for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py
[2018-09-10 15:49:26,993] {jobs.py:1108} INFO - No tasks to consider for execution.
[2018-09-10 15:49:27,001] {dag_processing.py:495} INFO - Processor for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py finished
[2018-09-10 15:49:28,009] {dag_processing.py:582} INFO - Started a process (PID: 29260) to generate tasks for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py
[2018-09-10 15:49:28,439] {jobs.py:1108} INFO - No tasks to consider for execution.
[2018-09-10 15:49:28,440] {dag_processing.py:495} INFO - Processor for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py finished
[2018-09-10 15:49:29,445] {dag_processing.py:582} INFO - Started a process (PID: 29261) to generate tasks for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py
[2018-09-10 15:49:29,872] {jobs.py:1108} INFO - No tasks to consider for execution.
[2018-09-10 15:49:29,873] {dag_processing.py:495} INFO - Processor for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py finished
[2018-09-10 15:49:30,876] {dag_processing.py:582} INFO - Started a process (PID: 29263) to generate tasks for /Users/xiang/Documents/BigData/airflow/dags/my_tutorial_2.py
[2018-09-10 15:49:31,309] {jobs.py:1108} INFO - No tasks to consider for execution.

Answer 1

调度程序将根据 airflow.cfg 的内容“检测”您的 dag 文件。可能与此最相关的两个设置是:

min_file_process_interval :文件解析循环之间要等待多少秒以防止日志被垃圾邮件发送。

scheduler_heartbeat_sec :调度程序不断尝试触发新任务(有关更多信息，请参阅文档中的调度程序部分)。这定义了调度程序应该运行的频率(以秒为单位)。

如果您只运行几个 DAG，并且任务不经常运行，请考虑更改这些设置。