kubernetes - 在 Kubernetes 的另一个脚本中运行 python 脚本的 Pod 中执行 bash 命令

Question

我正在使用本指南在我的 aks Kubernetes 集群中部署 pyspark:

https://towardsdatascience.com/ignite-the-spark-68f3f988f642

http://blog.brainlounge.de/memoryleaks/getting-started-with-spark-on-kubernetes/

如上面的链接中所述，我已经部署了我的驱动程序 pods :

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: spark
  name: my-notebook-deployment
  labels:
    app: my-notebook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-notebook
  template:
    metadata:
      labels:
        app: my-notebook
    spec:
      serviceAccountName: spark
      containers:
      - name: my-notebook
        image: pidocker-docker-registry.default.svc.cluster.local:5000/my-notebook:latest
        ports:
          - containerPort: 8888
        volumeMounts:
          - mountPath: /root/data
            name: my-notebook-pv
        workingDir: /root
        resources:
          limits:
            memory: 2Gi
      volumes:
        - name: my-notebook-pv
          persistentVolumeClaim:
            claimName: my-notebook-pvc
---
apiVersion: v1
kind: Service
metadata:
  namespace: spark
  name: my-notebook-deployment
spec:
  selector:
    app: my-notebook
  ports:
    - protocol: TCP
      port: 29413
  clusterIP: None

然后我可以使用以下代码创建 Spark 集群:

import os
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
# Create Spark config for our Kubernetes based cluster manager
sparkConf = SparkConf()
sparkConf.setMaster("k8s://https://kubernetes.default.svc.cluster.local:443")
sparkConf.setAppName("spark")
sparkConf.set("spark.kubernetes.container.image", "<MYIMAGE>")
sparkConf.set("spark.kubernetes.namespace", "spark")
sparkConf.set("spark.executor.instances", "7")
sparkConf.set("spark.executor.cores", "2")
sparkConf.set("spark.driver.memory", "512m")
sparkConf.set("spark.executor.memory", "512m")
sparkConf.set("spark.kubernetes.pyspark.pythonVersion", "3")
sparkConf.set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")
sparkConf.set("spark.kubernetes.authenticate.serviceAccountName", "spark")
sparkConf.set("spark.driver.port", "29413")
sparkConf.set("spark.driver.host", "my-notebook-deployment.spark.svc.cluster.local")
# Initialize our Spark cluster, this will actually
# generate the worker nodes.
spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
sc = spark.sparkContext

有用。
如何创建一个可以执行位于我的笔记本部署中的 python 脚本的外部 pod，我可以在我的终端中执行此操作:

kubectl exec my-notebook-deployment-7669bb6fc-29stw python3 myscript.py

但我希望能够在另一个 pod 中自动执行此命令

Answer 1

通常，您可以使用在其中运行的指定命令启动新 pod，即:

kubectl run mypod --image=python3 --command -- <cmd> <arg1> ... <argN>

在您的情况下，您需要将 myscript.py 的代码提供给 pod(即:通过安装带有脚本内容的 ConfigMap)或基于 python docker 构建新的容器镜像并将脚本添加到镜像中。