python - Pyspark，以编程方式初始化 spark : IllegalArgumentException: Missing application resource

Question

在 Python 中创建 spark 上下文时，出现以下错误。

 app_name="my_app"
 master="local[*]"
 sc = SparkContext(appName=app_name, master=master)

Exception in thread "main" java.lang.IllegalArgumentException: Missing application resource.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:241)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitArgs(SparkSubmitCommandBuilder.java:160)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:276)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:151)
at org.apache.spark.launcher.Main.main(Main.java:86)

....

pyspark.zip/pyspark/java_gateway.py", line 94, in launch_gateway
raise Exception("Java gateway process exited before sending the driver its port number")
Exception: Java gateway process exited before sending the driver its port number

Spark 发射器似乎以某种方式出现故障。

Answer 1

这是由于先前存在的环境变量发生冲突而发生的。我在 python 程序中删除了它们，现在它运行顺利。

例如:

import  os
#check if pyspark env vars are set and then reset to required or delete.   
del os.environ['PYSPARK_SUBMIT_ARGS']

正确的解决方案是在 .bashrc 或 .zshrc 或任何初始化它的 env 初始化脚本中删除它。但无法在 .bash_profile(mac)(.bashrc 或/etc/environment.conf)中找到它。如果找到位置，将更新答案