版本\"v1\"中的 KubernetesPodOperator Pod 不能作为 Pod : v1. Pod.Spec 处理

Question

使用 Airflow 在 AWS EKS 上运行 KubernetesPodOperator
获取 kubernetes.client.rest.ApiException: (400)，指出版本\"v1\"中的 Pod 不能作为 Pod 处理:v1.Pod.Spec。
aws-eks 不支持这种类型的 pod 创建吗？

这是完整的日志

[2019-03-13 01:13:47,991] {models.py:1593} INFO - Executing 
<Task(KubernetesPodOperator): failing-task> on 2019-03- 
13T01:12:33.980710+00:00
[2019-03-13 01:13:47,991] {base_task_runner.py:118} INFO - Running: 
['bash', '-c', 'airflow run testkuberete2 failing-task 2019-03- 
13T01:12:33.980710+00:00 --job_id 20 --raw -sd DAGS_FOLDER/test- 
kubepodopptest5.py --cfg_path /tmp/tmpmwb4uzk0']
[2019-03-13 01:13:48,415] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:48,415] {settings.py:174} INFO - 
settings.configure_orm(): Using pool settings. pool_size=5, 
pool_recycle=3600, pid=408
[2019-03-13 01:13:48,864] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:48,863] {__init__.py:51} INFO - 
Using executor CeleryExecutor
[2019-03-13 01:13:49,115] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:49,114] {models.py:273} INFO - 
Filling up the DagBag from /usr/local/airflow/dags/test- 
kubepodopptest5.py
[2019-03-13 01:13:49,473] {base_task_runner.py:101} INFO - Job 20: 
Subtask failing-task [2019-03-13 01:13:49,473] {cli.py:520} INFO - 
Running <TaskInstance: testkuberete2.failing-task 2019-03- 
13T01:12:33.980710+00:00 [running]> on host airflow-worker-1.airflow- 
worker.rguonew.svc.cluster.local

我写了一些自定义日志来检查发送到 EKS 的内容:

[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,781] {api_client.py:117} ERROR - header_params {Accept 
application/json}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,782] {api_client.py:117} ERROR - header_params {Content-Type 
application/json}
[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,782] {api_client.py:117} ERROR - header_params {User-Agent 
Swagger-Codegen/8.0.1/python}

 [2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
 01:13:49,782] {api_client.py:128} ERROR - resource_path {namespace 
 rguonew}

[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
   01:13:49,782] {api_client.py:151} ERROR - body {{'apiVersion': 'v1', 
   'kind': 'Pod', 'metadata': {'name': 'fail-422f98fc', 'labels': {}, 
    'annotations': {}}, 'spec': {'containers': [{'name': 'base', 
  'image': 
   'python:3.6.6-stretch', 'command': 'echo', 'imagePullPolicy': 
   'IfNotPresent', 'args': '10'}], 'restartPolicy': 'Never', 
   'nodeSelector': {}, 'volumes': [], 'serviceAccountName': 'default', 
    'affinity': {}}}}

[2019-03-13 01:13:49,782] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,782] {api_client.py:155} ERROR - url 
{https://xxxxxxxxxxxx.xxxx.us-east- 
1.eks.amazonaws.com/api/v1/namespaces/rguonew/pods}
[2019-03-13 01:13:49,838] {logging_mixin.py:95} INFO - [2019-03-13 
01:13:49,823] {pod_launcher.py:58} ERROR - Exception when attempting to 
create Namespaced Pod.

Traceback

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site- 
packages/airflow/contrib/kubernetes/pod_launcher.py", line 55, in 
run_pod_async
resp = self._client.create_namespaced_pod(body=req, 
namespace=pod.namespace)
File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/apis/core_v1_api.py", line 6115, in 
create_namespaced_pod
(data) = self.create_namespaced_pod_with_http_info(namespace, body, 
**kwargs)

File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/apis/core_v1_api.py", line 6206, in 
create_namespaced_pod_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content, 
_request_timeout)
File "/usr/local/lib/python3.6/site-packages/kubernetes/client/ap

i_client.py", line 163, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/api_client.py", line 372, in request
    body=body)
  File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/rest.py", line 266, in POST
    body=body)
  File "/usr/local/lib/python3.6/site- 
packages/kubernetes/client/rest.py", line 222, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request

回复:

HTTP response headers: HTTPHeaderDict({'Audit-Id': '96f8ffee-3e65-47a7- 
bb3c-fe519281ee45', 'Content-Type': 'application/json', 'Date': 'Wed, 
13 Mar 2019 01:13:49 GMT', 'Content-Length': '491'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata": 
{},"status":"Failure","message":"Pod in version \"v1\" cannot be 
handled as a Pod: v1.Pod.Spec: v1.PodSpec.Containers: []v1.Container: 
v1.Container.Command: []string: decode slice: expect [ or n, but found 
\", error found in #10 byte of ...|ommand\": \"echo\", \"im|..., bigger 
context ...|se\", \"image\": \"python:3.6.6-stretch\", \"command\": 
\"echo\", \"imagePullPolicy\": \"IfNotPresent\", \"args\": 
|...","reason":"BadRequest","code":400}
[2019-03-13 01:13:49,838] {logging_mixin.py:95} INFO - 

Answer 1

啊，很高兴找到另一个在 EKS 上使用 Airflow 的人!!

我也在运行类似的 Airflow 工作负载，可以确认 AWS EKS 支持 KubernetesPodOperator。

查看响应，我认为在实例化 KubernetesPodOperator 时，用于创建 pod 的 API 调用可能被破坏，或者您的 Python 代码包含不正确的参数类型。

我解码了您的 pod 定义(如果有误，请纠正我):

---
apiVersion: v1
kind: Pod
metadata:
  name: fail-422f98fc
  labels: {}
  annotations: {}
spec:
  containers:
  - name: base
    image: python:3.6.6-stretch
    command: echo
    imagePullPolicy: IfNotPresent
    args: '10'
  restartPolicy: Never
  nodeSelector: {}
  volumes: []
  serviceAccountName: default
  affinity: {}

您应该将“命令”和“参数”括在方括号中，如下所示:

---
apiVersion: v1
kind: Pod
metadata:
  name: fail-422f98fc
  labels: {}
  annotations: {}
spec:
  containers:
  - name: base
    image: python:3.6.6-stretch
    command: ['echo']
    imagePullPolicy: IfNotPresent
    args: ['10']
  restartPolicy: Never
  nodeSelector: {}
  volumes: []
  serviceAccountName: default
  affinity: {}

希望这可以帮助!