scalability - 如何解决 Argo 输出参数大小限制？

Question

我有一个遍历 JSON 数组的 Argo 工作流。当列表变得太大时，我会收到如下错误:

time="some-time" level=fatal msg="Pod \"some-pod-name\" is invalid: metadata.annotations: Too long: must have at most 262144 characters"

或者， in newer versions of Argo :

Output is larger than the maximum allowed size of 256 kB, only the last 256 kB were saved

如何在不达到大小限制的情况下遍历这个大型 JSON 数组？
我的工作流程看起来有点像这样，但有一个更大的 JSON 数组:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: item
            value: "{{item}}"
        withParam: "{{steps.get-items.outputs.parameters.items}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items"]
    outputs:
      parameters:
      - name: items
        valueFrom:
          path: /tmp/items
  - name: echo
    inputs:
      parameters:
      - name: item
    container:
      image: stedolan/jq:latest
      command: [echo, "{{inputs.parameters.item}}"]

Answer 1

不是将 JSON 数组写入输出参数，而是将其写入工件并将其长度写入输出参数。然后您可以使用 withSequence 循环遍历数组索引并从 JSON 工件中检索相应的项目。
这是一个带有硬编码 JSON 数组和项目计数的示例。您的 get-items 步骤肯定会更复杂。

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-sequence-
spec:
  entrypoint: loops-sequence
  templates:
  - name: loops-sequence
    steps:
    - - name: get-items
        template: get-items
    - - name: sequence-param
        template: echo
        arguments:
          parameters:
          - name: index
            value: "{{item}}"
          artifacts:
          - name: items
            from: "{{steps.get-items.outputs.artifacts.items}}"
        withSequence:
          count: "{{steps.get-items.outputs.parameters.count}}"
  - name: get-items
    container:
      image: alpine:latest
      command: ["/bin/sh", "-c"]
      args: ["echo '[\"a\", \"b\", \"c\"]' > /tmp/items && echo '3' > /tmp/count"]
    outputs:
      artifacts:
      - name: items
        path: /tmp/items
      parameters:
      - name: count
        valueFrom:
          path: /tmp/count
  - name: echo
    inputs:
      parameters:
      - name: index
      artifacts:
      - name: items
        path: /tmp/items
    container:
      image: stedolan/jq:latest
      command: [sh, -c]
      args: ["cat /tmp/items | jq '.[{{inputs.parameters.index}}]'"]