apache-spark - 用 kubernetes 代替 yarn 激发 Spark

Question

我正在编写一个使用 kubernetes 而不是 yarn 的 spark 作业。

val spark = SparkSession.builder().appName("Demo").master(????).getOrCreate() 

那么我的master应该是什么，是kubernetes还是其他的

谢谢

Answer 1

我的格式应该是k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port>

来自文档 here

URL with the format k8s://. Prefixing the master string with k8s:// will cause the Spark application to launch on the Kubernetes cluster, with the API server being contacted at api_server_url. If no HTTP protocol is specified in the URL, it defaults to https. For example, setting the master to k8s://example.com:443 is equivalent to setting it to k8s://https://example.com:443, but to connect without TLS on a different port, the master would be set to k8s://http://example.com:8080.