python - 在Docker上运行时，Airflow提供日志文件不存在错误

Question

调度程序和Web服务器正在不同的容器上运行，当我运行DAG并检查Web服务器上的日志时，它向我显示了此特定错误。

*** Log file does not exist: /usr/local/airflow/logs/indexing/index_articles/2019-12-31T00:00:00+00:00/1.log
*** Fetching from: http://465e0f4a4332:8793/log/indexing/index_articles/2019-12-31T00:00:00+00:00/1.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='465e0f4a4332', port=8793): Max retries exceeded with url: /log/indexing/index_articles/2019-12-31T00:00:00+00:00/1.log (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0a143700d0>: Failed to establish a new connection: [Errno 111] Connection refused'))

我设置了另一个类似问题中提到的 Airflow 变量，而我在cfg文件上更改的唯一变量是这些变量。

AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
AIRFLOW__CORE__LOAD_EXAMPLES=False
AIRFLOW__CORE__BASE_URL = http://{hostname}:8080

我手动检查并正确生成了日志文件，我假设唯一的问题是URL无法通过Web服务器容器公开访问。我不确定我在哪里弄乱了，我正在本地运行和测试它。

Answer 1

worker 的主机名未正确解析。
添加文件hostname_resolver.py:

import os
import socket
import requests
def resolve():
    """
    Resolves Airflow external hostname for accessing logs on a worker
    """
    if 'AWS_REGION' in os.environ:
        # Return EC2 instance hostname:
        return requests.get(
            'http://169.254.169.254/latest/meta-data/local-ipv4').text
    # Use DNS request for finding out what's our external IP:
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.connect(('1.1.1.1', 53))
    external_ip = s.getsockname()[0]
    s.close()
    return external_ip

并导出: AIRFLOW__CORE__HOSTNAME_CALLABLE=airflow.hostname_resolver:resolve