gpt4 book ai didi

python - 在 Kubeflow 管道中共享 secret

转载 作者:行者123 更新时间:2023-12-05 04:52:38 25 4
gpt4 key购买 nike

我想与我的 Kubeflow 管道分享一些 secret ,这样我就可以将它们用作容器中的环境变量。我写了一个 pipeline-secrets.yaml,看起来像这样:

apiVersion: v1
kind: Secret
metadata:
name: pipeline-secrets
namespace: kubeflow
type: Opaque
data:
mysql_db_name: <SECRET>
mysql_username: <SECRET>
mysql_password: <SECRET>
mysql_endpoint: <SECRET>

和一个 pipeline-pod-defaults.yaml 看起来像这样:

apiVersion: kubeflow.org/v1alpha1
kind: PodDefault
metadata:
name: pipeline-pod-defaults
namespace: kubeflow
specs:
desc: Configure pipeline secrets as environment variables
env:
- name: MYSQL_DB_NAME
valueFrom:
secretKeyRef:
name: pepeline-secrets
key: mysql_db_name
- name: MYSQL_USER_NAME
valueFrom:
secretKeyRef:
name: pepeline-secrets
key: mysql_username
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
name: pepeline-secrets
key: mysql_password
- name: MYSQL_ENDPOINT
valueFrom:
secretKeyRef:
name: pepeline-secrets
key: mysql_endpoint

这就是我的管道的样子:

import kfp
from kfp.dsl import ContainerOp
from kubernetes import client as k8s_client

@kfp.dsl.pipeline(
name="Training pipeline",
description=""
)
def train_pipeline():
get_data = ContainerOp(
name="Get data",
image=BASE_IMAGE,
file_outputs={
'data': 'data.csv'
}
)

kfp.dsl.get_pipeline_conf().set_image_pull_secrets([
k8s_client.V1ObjectReference(name="regcred"),
k8s_client.V1ObjectReference(name="pipeline-secrets"),
])
kfp.dsl.ResourceOp(
name="pipeline-pod-defaults",
k8s_resource=k8s_client.V1ObjectReference(name="pipeline-pod-defaults"),
action="apply"
)

但最后我得到了这个错误:

This step is in Failed state with this message: error: error validating "/tmp/manifest.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false

这是正确的做法吗?我如何与管道的其余部分分享我的 secret ?抱歉,如果这是一个新手问题,我是 Kubernetes 和 Kubeflow 的新手

最佳答案

所以,最后,我所做的是编写一个 get-data-components.yaml 来创建我的组件并编写下面的函数并且它起作用了:

def build_get_data():
component = kfp.components.load_component_from_file(os.path.join(COMPONENTS_PATH, 'get-data-component.yaml'))()
component.add_volume(k8s_client.V1Volume(
name="get-data-volume",
secret=k8s_client.V1SecretVolumeSource(secret_name="pipeline-secrets"))
)
envs = [
("MYSQL_DB_NAME", "mysql_db_name"),
("MYSQL_USER_NAME", "mysql_username"),
("MYSQL_PASSWORD", "mysql_password"),
("MYSQL_ENDPOINT", "mysql_endpoint")
]
for name, key in envs:
component.add_env_variable(
V1EnvVar(
name=name,
value_from=k8s_client.V1EnvVarSource(secret_key_ref=k8s_client.V1SecretKeySelector(
name="pipeline-secrets",
key=key
)
)
)
)
return component

关于python - 在 Kubeflow 管道中共享 secret ,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/66423449/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com