airflow - 在哪里查看 Airflow 作业的输出

Question

在 Airflow 教程中，BashOperator 有输出(通过 echo)。如果任务在调度程序中运行，您在哪里查看输出？有控制台什么的吗？我确信我只是没有找对地方。

Answer 1

就像 @tobi6 所说，您可以根据环境在网络服务器或控制台中查看 DAG 运行的输出。

要在您的网络服务器中执行此操作:

1. 选择您刚刚运行的 DAG 并进入Graph View。
2. 选择该 DAG 中您要查看其输出的任务。
3. 在下面的弹出窗口中，点击查看日志。
4. 在以下日志中，您现在可以看到输出，或者它会为您提供指向可以查看输出的页面的链接(例如，如果您使用的是 Databricks，最后一行可能是“INFO - 查看运行状态” 、Spark UI 和日志位于domain.cloud.databricks.com#job/jobid/run/1")。

如果您想查看运行日志，请在 airflow_home 目录中进行操作。

• 以下日志中的信息来自 Airflow 官方文档:

Users can specify a logs folder in airflow.cfg. By default, it is in the AIRFLOW_HOME directory.

In addition, users can supply a remote location for storing logs and log backups in cloud storage. At this time, Amazon S3 and Google Cloud Storage are supported. To enable this feature, airflow.cfg must be configured as in this example:

[core]
# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
# must supply a remote location URL (starting with either 's3://...' or
# 'gs://...') and an Airflow connection id that provides access to the storage
# location.
remote_base_log_folder = s3://my-bucket/path/to/logs
remote_log_conn_id = MyS3Conn
# Use server-side encryption for logs stored in S3
encrypt_s3_logs = False
Remote logging uses an existing Airflow connection to read/write logs. If you don’t have a connection properly setup, this will fail.

In the above example, Airflow will try to use S3Hook('MyS3Conn').

In the Airflow Web UI, local logs take precedance over remote logs. If local logs can not be found or accessed, the remote logs will be displayed. Note that logs are only sent to remote storage once a task completes (including failure). In other words, remote logs for running tasks are unavailable. Logs are stored in the log folder as {dag_id}/{task_id}/{execution_date}/{try_number}.log.