hadoop - spark-submit 如何设置user.name

Question

要设置

mapreduce.job.user.name=myuser

尝试过

spark-submit  --class com.MyClass
--conf mapreduce.job.user.name=myuser \ 
--conf spark.mapreduce.job.user.name=myuser \
--master yarn  \
--deploy-mode cluster \

也试过

--conf user.name

在 Spark UI 环境中显示

user.name yarn

Answer 1

在 Spark 3 中，将 SPARK_USER 设置为系统属性。

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala

  /**
   * Returns the current user name. This is the currently logged in user, unless that's been
   * overridden by the `SPARK_USER` environment variable.
   */
  def getCurrentUserName(): String = {
    Option(System.getenv("SPARK_USER"))
      .getOrElse(UserGroupInformation.getCurrentUser().getShortUserName())
  }