python - Airflow CeleryExecutor - 'int' 对象在 Celery 中没有属性 'startswith'

Question

Airflow 2.0 正在排队，但未在我的开发环境中启动任务。

DAG 和 Pool 设置是有效的，但是每个 dag 中的所有任务在我触发它们时都排队，并且永远不会运行。

输入 airflow celery worker 时出现以下错误:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/celery_command.py", line 188, in worker
    _run_worker(options=options, skip_serve_logs=skip_serve_logs)
  File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/celery_command.py", line 94, in _run_worker
    celery_app.worker_main(options)
  File "/usr/local/lib/python3.8/site-packages/celery/app/base.py", line 365, in worker_main
    return instantiate(
  File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 283, in execute_from_commandline
    self.maybe_patch_concurrency(argv)
  File "/usr/local/lib/python3.8/site-packages/celery/bin/base.py", line 315, in maybe_patch_concurrency
    maybe_patch_concurrency(argv, *pool_option)
  File "/usr/local/lib/python3.8/site-packages/celery/__init__.py", line 143, in maybe_patch_concurrency
    pool = _find_option_with_arg(argv, short_opts, long_opts)
  File "/usr/local/lib/python3.8/site-packages/celery/__init__.py", line 95, in _find_option_with_arg
    if arg.startswith('-'):
AttributeError: 'int' object has no attribute 'startswith'

在我的临时生产环境中，运行任务以及检查 airflow celery worker 都没有问题。它会启动或警告我它已经在运行(如预期的那样)。

env 之间没有区别，但我猜是最近在服务器上部署后出现的问题。

如我所见， celery 收到了错误的论点:

AttributeError: 'int' object has no attribute 'startswith'

但是如何追踪哪些参数 Airflow 试图传递给 celery？我不知道如何调试。

Answer 1

通过将 celery 升级到最新版本从 4.4.2 到 5.1.2 解决。

似乎版本 4.4.2(它是 Airflow 部门之一)有参数错误。

如果有任何其他解决此问题的建议，请随时在此处提及。