scala - Scala-Spark 开发人员是否必须在他的计算机上安装 Spark 和 Hadoop？

Question

我已经在服务器上安装了 Hadoop + Spark 集群。
在主服务器上的 spark-shell 中编写 scala 代码可以正常工作。
我将 Spark 库(jar 文件)放在我的项目中，并通过 Intellij 在我的计算机上编写我的第一个 Scala 代码。
当我运行一个简单的代码来创建一个 SparkContext 对象以通过 hdfs 协议(protocol)从 HDFS 读取文件时，它会输出错误消息。
测试功能:

import org.apache.spark.SparkContext

class SpcDemoProgram {

  def demoPrint(): Unit ={
    println("class spe demoPrint")
    test()
  }

  def test(){

    var spark = new SparkContext();
  }
}

消息是:

20/11/02 12:36:26 INFO SparkContext: Running Spark version 3.0.020/11/02 12:36:26 WARN Shell: Did not find winutils.exe: {}java.io.FileNotFoundException: java.io.FileNotFoundException:HADOOP_HOME and hadoop.home.dir are unset. -seehttps://wiki.apache.org/hadoop/WindowsProblems atorg.apache.hadoop.util.Shell.fileNotFoundException(Shell.java:548) atorg.apache.hadoop.util.Shell.getHadoopHomeDir(Shell.java:569) atorg.apache.hadoop.util.Shell.getQualifiedBin(Shell.java:592) atorg.apache.hadoop.util.Shell.(Shell.java:689) atorg.apache.hadoop.util.StringUtils.(StringUtils.java:78) atorg.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1664)atorg.apache.hadoop.security.SecurityUtil.setConfigurationInternal(SecurityUtil.java:104)atorg.apache.hadoop.security.SecurityUtil.(SecurityUtil.java:88)atorg.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:316)atorg.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:304)atorg.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1828)atorg.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)atorg.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)atorg.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)atorg.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2412)at scala.Option.getOrElse(Option.scala:189) atorg.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2412) atorg.apache.spark.SparkContext.(SparkContext.scala:303) atorg.apache.spark.SparkContext.(SparkContext.scala:120) atscala.spc.demo.SpcDemoProgram.test(SpcDemoProgram.scala:14) atscala.spc.demo.SpcDemoProgram.demoPrint(SpcDemoProgram.scala:9) atscala.spc.demo.SpcDemoProgram$.main(SpcDemoProgram.scala:50) atscala.spc.demo.SpcDemoProgram.main(SpcDemoProgram.scala) Caused by:java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir areunset. atorg.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:468) atorg.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:439) atorg.apache.hadoop.util.Shell.(Shell.java:516) ... 19 more20/11/02 12:36:26 WARN NativeCodeLoader: Unable to load native-hadooplibrary for your platform... using builtin-java classes whereapplicable 20/11/02 12:36:27 ERROR SparkContext: Error initializingSparkContext. org.apache.spark.SparkException: A master URL must beset in your configuration atorg.apache.spark.SparkContext.(SparkContext.scala:380) atorg.apache.spark.SparkContext.(SparkContext.scala:120) atscala.spc.demo.SpcDemoProgram.test(SpcDemoProgram.scala:14) atscala.spc.demo.SpcDemoProgram.demoPrint(SpcDemoProgram.scala:9) atscala.spc.demo.SpcDemoProgram$.main(SpcDemoProgram.scala:50) atscala.spc.demo.SpcDemoProgram.main(SpcDemoProgram.scala) 20/11/0212:36:27 INFO SparkContext: Successfully stopped SparkContextException in thread "main" org.apache.spark.SparkException: A masterURL must be set in your configuration atorg.apache.spark.SparkContext.(SparkContext.scala:380) atorg.apache.spark.SparkContext.(SparkContext.scala:120) atscala.spc.demo.SpcDemoProgram.test(SpcDemoProgram.scala:14) atscala.spc.demo.SpcDemoProgram.demoPrint(SpcDemoProgram.scala:9) atscala.spc.demo.SpcDemoProgram$.main(SpcDemoProgram.scala:50) atscala.spc.demo.SpcDemoProgram.main(SpcDemoProgram.scala)

该错误消息是否暗示必须在我的计算机上安装 Hadoop 和 Spark？
我需要做什么配置？

Answer 1

我假设，您正在尝试读取路径为 hdfs://<FILE_PATH> 的文件那么是的，如果它只是一个本地目录，您需要安装 Hadoop，您可以尝试在文件路径中不使用“hdfs://”。