python - 如何在Google CloudBuild步骤中保留变量？

Question

我有一个cloudbuild.json用于将管道上传到gcp kubeflow。现在，我想添加另一个步骤，以获取最新的管道ID，然后将其作为实验运行。
所以我的主要问题是如何在后续步骤中获取管道ID。我已经编写了一个小脚本来获取最新的管道ID，并将其添加为从Docker运行的步骤，但是现在我不确定如何获取此管道ID。
这是我的coudbuild.json

{
    "steps": [
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "trainer_image",
                "."
            ],
            "dir": "./trainer_image/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "base_image",
                "."
            ],
            "dir": "./base_image/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "dsl-compile --py covertype_training_pipeline.py --output covertype_training_pipeline.yaml"
            ],
            "env": [
                "BASE_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test",
                "TRAINER_IMAGE=gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
                "RUNTIME_VERSION=1.15",
                "PYTHON_VERSION=3.7",
                "COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/",
                "USE_KFP_SA=False"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/dmgcp-pkg-internal-poc-oct-04/kfp-cli",
            "args": [
                "-c",
                "kfp --endpoint 66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com pipeline upload -p credit_fraud_training_test covertype_training_pipeline.yaml"
            ],
            "dir": "./pipeline/"
        },
        {
            "name": "gcr.io/cloud-builders/docker",
            "args": [
                "build",
                "-t",
                "id_image",
                "."
            ],
            "dir": "./id_image/"
        }
    ],
    "images": [
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/trainer_image:test",
        "gcr.io/dmgcp-pkg-internal-poc-oct-04/base_image:test"
    ]
}

这是我的python脚本来获取最新的管道ID

import kfp
client = kfp.Client(host='66df1d31e46e6510-dot-us-central2.pipelines.googleusercontent.com')
pipelines = client.list_pipelines()
total_pipeline = len(pipelines.pipelines)
latest_pipeline_id = pipelines.pipelines[total_pipeline-1].id
with open('/workspace/pipeline.txt') as file:
    file.write(latest_pipeline_id)

Answer 1

您不能在步骤之间保留变量。您只能保留文件。但是，在一个步骤中，您可以重用局部变量。

- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(docker version)
      echo $$DOCKER_VERSION #Reuse local variable
      echo $$DOCKER_VERSION > DOCKER_VERSION.txt #Save the variable for the next step
- name: "gcr.io/cloud-builders/docker",
  entrypoint: "bash"
  args: 
   - "-c"
   - |
      export DOCKER_VERSION=$(cat DOCKER_VERSION.txt) #Get the value saved in file
      echo $$DOCKER_VERSION #Reuse local variable