java - scala mapreduce 异常 : java. lang.ClassNotFoundException : scala. Function2

Question

我已经在我自己的 ubuntu linux18.04 机器上安装并配置了 jdk 1.8/hadoop 2.8.4/scala 2.10.6，WordCount java 应用程序使用“hadoop jar”命令运行正常。

然后我在与 java wordcount 相同的 intellij 项目中尝试了 scala 代码，代码如下:

import java.io.IOException
import java.util._

import org.apache.hadoop.fs.Path
import org.apache.hadoop.io._
import org.apache.hadoop.mapred._

object wc01 {
@throws[Exception]
def main(args: Array[String]) {
    val conf: JobConf = new JobConf(this.getClass)
    conf.setJobName("WordCountScala")
    conf.setOutputKeyClass(classOf[Text])
    conf.setOutputValueClass(classOf[IntWritable])
    conf.setMapperClass(classOf[Map])
    conf.setCombinerClass(classOf[Reduce])
    conf.setReducerClass(classOf[Reduce])
    conf.setInputFormat(classOf[TextInputFormat])
    conf.setOutputFormat(classOf[TextOutputFormat[Text, IntWritable]])
    FileInputFormat.setInputPaths(conf, new Path(args(0)))
    FileOutputFormat.setOutputPath(conf, new Path(args(1)))
    JobClient.runJob(conf)
}

class Map extends MapReduceBase with Mapper[LongWritable, Text, Text, IntWritable] {
    private final val one = new IntWritable(1)
    private val word = new Text()

    @throws[IOException]
    def map(key: LongWritable, value: Text, output: OutputCollector[Text, IntWritable], reporter: Reporter) {
    val line: String = value.toString
    line.split(" ").foreach { token =>
        word.set(token)
        output.collect(word, one)
    }
    }
}

class Reduce extends MapReduceBase with Reducer[Text, IntWritable, Text, IntWritable] {
    @throws[IOException]
    def reduce(key: Text, values: Iterator[IntWritable], output: OutputCollector[Text, IntWritable], reporter: Reporter) {
    import scala.collection.JavaConversions._
    val sum = values.toList.reduce((valueOne, valueTwo) => new IntWritable(valueOne.get() + valueTwo.get()))
    output.collect(key, new IntWritable(sum.get()))
    }
}
}

我编译打包，用hadoop jar运行，报错:

hdfs@ubuntu:$ hadoop jar my_java_scala_mr-1.0-SNAPSHOT.jar wc01 my-input my-output
18/08/26 01:30:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/08/26 01:30:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/08/26 01:30:58 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
18/08/26 01:30:58 INFO mapred.FileInputFormat: Total input files to process : 1
18/08/26 01:30:58 INFO mapreduce.JobSubmitter: number of splits:2
18/08/26 01:30:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1535165327468_0012
18/08/26 01:30:59 INFO impl.YarnClientImpl: Submitted application application_1535165327468_0012
18/08/26 01:30:59 INFO mapreduce.Job: The url to track the job: http://ubuntu:8088/proxy/application_1535165327468_0012/
18/08/26 01:30:59 INFO mapreduce.Job: Running job: job_1535165327468_0012
18/08/26 01:31:04 INFO mapreduce.Job: Job job_1535165327468_0012 running in uber mode : false
18/08/26 01:31:04 INFO mapreduce.Job:  map 0% reduce 0%
18/08/26 01:31:08 INFO mapreduce.Job: Task Id : attempt_1535165327468_0012_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: scala.Function2
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

我想知道我是否需要任何额外的 hadoop java 包来支持 scala MR？我没有在我的 pom.xml 中指定任何自定义包语句，我只是“mvn package”来生成我的 jar，看起来没问题。

我该如何解决这个问题？

Answer 1

听起来您缺少 Scala 标准库。尝试将 org.scala-lang/scala-library/2.12.6 添加到您的依赖项中。