postgresql - Airflow - 为什么外部数据库配置会破坏 helm 升级？

Question

我正在尝试使用 Helm 图表部署 Airflow，以进行个人 POC，但我在部署方面遇到了一些问题，并且找不到明确的说明来解决我的问题 - 这就是我在这里寻求帮助的原因。

问题的背景

首先了解一下 POC 的背景 - 我想部署一个托管 Airflow 的 K8S 集群，将其连接到托管 dags 的 git 存储库并托管元存储和缓存外部来自 K8S Airflow。

我已使用 kind 和 Airflow 的默认 Helm Chart 成功将 Airflow 部署到本地 Kubernetes 集群。在 helm 图表上，我指定要使用的执行器模式必须是 KubernetesExecutor。

我还配置了 Airflow，以将 DAG 同步到 bitbucket 存储库或从 bitbucket 存储库同步。

问题和当前实现

我在将 Airflow 与外部服务连接时遇到问题 - 我创建了一个 Azure PostgreSQL 服务器，创建了一个 Airflow 数据库，并在 psql 上创建了一个管理员用户，如下所示:

CREATE DATABASE airflow;
CREATE USER aflw_admin WITH PASSWORD 'some_password';
GRANT ALL PRIVILEGES ON DATABASE airflow TO aflw_admin;
ALTER USER aflw_admin SET search_path = public;

由于我使用 helm 进行部署，因此我的 values.yaml 如下:

postgresql:
  enabled: false 

externalDatabase:
  type: postgres
  host: dbname.postgres.database.azure.com
  port: 5432
  database: airflow
  user: aflw_admin
  passwordSecretKey: "postgresql-password"

data:
  metadataSecretName: ~
  resultBackendSecretName: ~

  metadataConnection:
    user: aflw_admin 
    pass:  some_password
    protocol: postgresql
    host: dbname.postgres.database.azure.com
    port: 5432
    db: airflow 
    sslmode: require 
    
  resultBackendConnection:
    user: aflw_admin 
    pass:  some_password
    protocol: postgresql
    host: dbname.postgres.database.azure.com
    port: 5432
    db: airflow 
    sslmode: require 

secret postgresql-password 是由以下内容创建的:

kubectl create secret generic airflow-postgresql --from-literal=postgresql-password=$(openssl rand -base64 13) --namespace airflow

我使用以下方法部署了解决方案:

kubectl apply -f ./helm/variables.yaml
helm upgrade --install airflow apache-airflow/airflow -n airflow -f ./values.yaml --debug 

我尝试过的内容和问题详细信息

经过一番来回，我发现通过恢复配置 - 也就是将 postgresql 设置为 true 并删除 metadataConnection values.yaml 文件中的 resultBackendConnection 和 externalDatabase 部分 - 我可以成功部署 postgres 服务，但需要权衡 postgresql不是外部服务，这有助于至少部分隔离问题。

因此，如果我返回初始配置并尝试部署它，我得到的结果是:

• 首先我得到一个超时 --> 为了面对这个问题，我自然地将超时持续时间增加到一个更大的值，例如20m0s;
• 增加超时后出现错误 BackoffLimitExceeded 并且没有部署任何内容。

以下是相关 helm 部署的日志:

history.go:56: [debug] getting history for release airflow
upgrade.go:142: [debug] preparing upgrade for airflow
upgrade.go:150: [debug] performing update for airflow
upgrade.go:322: [debug] creating upgraded release for airflow
client.go:218: [debug] checking 20 resources for changes
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-create-user-job"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-migrate-database-job"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-scheduler"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-statsd"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-triggerer"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-webserver"
client.go:501: [debug] Looks like there are no changes for ServiceAccount "airflow-worker"
client.go:501: [debug] Looks like there are no changes for Secret "airflow-airflow-metadata"
client.go:501: [debug] Looks like there are no changes for Secret "airflow-webserver-secret-key"
client.go:501: [debug] Looks like there are no changes for ConfigMap "airflow-airflow-config"
client.go:501: [debug] Looks like there are no changes for Role "airflow-pod-launcher-role"
client.go:501: [debug] Looks like there are no changes for Role "airflow-pod-log-reader-role"
client.go:501: [debug] Looks like there are no changes for RoleBinding "airflow-pod-launcher-rolebinding"
client.go:501: [debug] Looks like there are no changes for RoleBinding "airflow-pod-log-reader-rolebinding"
client.go:501: [debug] Looks like there are no changes for Service "airflow-statsd"
client.go:501: [debug] Looks like there are no changes for Service "airflow-webserver"
client.go:510: [debug] Patch Deployment "airflow-scheduler" in namespace airflow
client.go:510: [debug] Patch Deployment "airflow-statsd" in namespace airflow
client.go:510: [debug] Patch Deployment "airflow-triggerer" in namespace airflow
client.go:510: [debug] Patch Deployment "airflow-webserver" in namespace airflow
client.go:267: [debug] Deleting Secret "airflow-postgresql" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql", err: secrets "airflow-postgresql" not found
client.go:267: [debug] Deleting Service "airflow-postgresql-headless" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql-headless", err: services "airflow-postgresql-headless" not found
client.go:267: [debug] Deleting Service "airflow-postgresql" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql", err: services "airflow-postgresql" not found
client.go:267: [debug] Deleting StatefulSet "airflow-postgresql" in namespace airflow...
client.go:270: [debug] Unable to get obj "airflow-postgresql", err: statefulsets.apps "airflow-postgresql" not found
client.go:299: [debug] Starting delete for "airflow-run-airflow-migrations" Job
client.go:128: [debug] creating 1 resource(s)
client.go:529: [debug] Watching for changes to Job airflow-run-airflow-migrations with timeout of 20m0s
client.go:557: [debug] Add/Modify event for airflow-run-airflow-migrations: ADDED
client.go:596: [debug] airflow-run-airflow-migrations: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:557: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
client.go:596: [debug] airflow-run-airflow-migrations: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:557: [debug] Add/Modify event for airflow-run-airflow-migrations: MODIFIED
upgrade.go:433: [debug] warning: Upgrade "airflow" failed: post-upgrade hooks failed: job failed: BackoffLimitExceeded
Error: UPGRADE FAILED: post-upgrade hooks failed: job failed: BackoffLimitExceeded
helm.go:84: [debug] post-upgrade hooks failed: job failed: BackoffLimitExceeded
UPGRADE FAILED
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:199
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="b4d7dbd6c6d5f4c2859a879a84" rel="noreferrer noopener nofollow">[email protected]</a>/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="20434f4252416056110e130e10" rel="noreferrer noopener nofollow">[email protected]</a>/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="85e6eae7f7e4c5f3b4abb6abb5" rel="noreferrer noopener nofollow">[email protected]</a>/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581
make: *** [Makefile:46: deploy-airflow] Error 1

这种行为让我认为这是某种配置错误，但我无法确定是什么。

我的 Helm 图表中的哪些配置错误可能会破坏 Helm 升级？

helm/airflow/psql 的版本如下:

• Airflow -> apache/airflow:2.2.3
• Helm 图表 -> 版本 1.4.0 的默认图像 ( https://artifacthub.io/packages/helm/apache-airflow/airflow )
• PSQL(在 Azure 上) -> Azure Database for PostgreSQL 灵活服务器，PSQL 版本 13.4

Answer 1

有时很难诊断此类问题，因为有太多移动部件。不过，我已经在 Azure AKS(Postgres sslmode:需要)和 AWS EKS(RDS Postgres sslmode:禁用)上设置了 Airflow，每个都有自己的问题。

也许删除externalDatabase和resultBackendConnection的配置。为什么？ - 因为如果未配置，resultBackendConnection 将使用metadataConnection。我在当前的配置文件 v2.2.4 中没有看到 externalDatabase 键。您是否使用 -f Values.yaml 使用正确的 Values.yaml 覆盖 helm 安装？

如果禁用 postgresql

postgresql:
  enabled: false

如您所愿，您需要为外部数据库配置metadataConnection。

只有在与 Postgres 的连接正常工作后，我才配置了 metadataSecretName。

另外，请尝试在metadataConnection配置中禁用sslmode:disable。

一旦我按照我想要的方式获得了配置文件，我就卸载了 Airflow，然后重新安装:

1. Helm 删除 Airflow
2. 删除了 Airflow 数据库并重新创建了
3. kubectl 删除 secret [airflow-xxxxx-xxxxx] 中的所有 secret 命名空间，因为数据库迁移在错误。
4. kubectl delete pvc(然后确保 pv 也被删除)

之后我重新安装，一切都很好，这不是很多工作，但确保我可以使用正确的值重新部署。

哦，请记住设置 PSQL 并确保您实际上可以从命令行进行连接，作为额外检查。