gpt4 book ai didi

scala - Hadoop 作业在 java.lang.ClassNotFoundException 上失败

转载 作者:可可西里 更新时间:2023-11-01 15:03:07 24 4
gpt4 key购买 nike

我已经搜索过这个问题的解决方案,但无济于事。我有三个 Scala 类:MaxTemperatureMapperMaxTemperatureReducerMaxTemperatureDriver(请参阅下面的实现)。之前线程中的一些人删除了 Main 类 (MaxTemperatureDriver) 中的 job.setJar() 方法以使 hadoop 作业运行。那对我不起作用。运行时我继续收到以下堆栈跟踪:

  1. hadoop com.koadr.hadoop.MaxTemperatureDriver micro/sample.txt 输出

  2. hadoop jar target/classes/koadr-hadoop-1.0-SNAPSHOT.jar com.koadr.hadoop.MaxTemperatureDriver micro/sample.txt 输出:

.

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.koadr.hadoop.MaxTemperatureMapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1587)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:631)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: Class com.koadr.hadoop.MaxTemperatureMapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1585)
... 8 more

我在一个 intellij IDE 中工作,并使用 maven 进行依赖、编译等。我在命令行上运行上述 hadoop 命令。有人可以解释我哪里出错了吗?为什么找不到 Mapper 类?

最大温度映射器:

class MaxTemperatureMapper extends Mapper[Object, Text, Text, IntWritable] {

private def missing( temp : String) : Boolean = {
temp.equals("+9999")
}

override def map(key: Object, value: Text, context:Mapper[Object,Text,Text,IntWritable]#Context ) = {
val line : String = value.toString
val year : String = line.substring(15, 19)
val temp : String = line.substring(87,92)


if (!missing(temp)) {
val airTemp : Int = Integer.parseInt(line.substring(87,92))
context.write(new Text(year), new IntWritable(airTemp))
}
}
}

最大温度降低器:

class MaxTemperatureReducer extends Reducer[Text, IntWritable, Text, IntWritable] {

override
def reduce (key: Text, values: java.lang.Iterable[IntWritable], context:Reducer[Text, IntWritable, Text, IntWritable]#Context) = {
def maxVal(values : List[IntWritable], cMaxV : Int) : Int = {
if (values.isEmpty) cMaxV
else maxVal(values.tail,math.max(cMaxV, values.head.get()))
}
context write(new Text(key), new IntWritable(maxVal(values.toList,Integer.MIN_VALUE)) )
}

}

最大温度驱动器:

class MaxTemperatureDriver extends Configured with Tool {
override
def run(args : Array[String]) : Int = {
if (args.length != 2) {
System.err.printf("Usage: %s [generic options] <input> <output>\n", getClass.getSimpleName)
ToolRunner.printGenericCommandUsage(System.err)
-1
}
val job : Job = Job.getInstance(getConf, "Max Temperature")
job.setJarByClass(getClass)

FileInputFormat.addInputPath(job, new Path(args(0)))
FileOutputFormat.setOutputPath(job, new Path(args(1)))

job.setMapperClass(classOf[MaxTemperatureMapper])
job.setCombinerClass(classOf[MaxTemperatureReducer])
job.setReducerClass(classOf[MaxTemperatureReducer])

job.setOutputKeyClass(classOf[Text])
job.setOutputValueClass(classOf[IntWritable])

if (job.waitForCompletion(true)) 0 else 1

}


}

object MaxTemperatureDriver {
def main(args : Array[String]) = {
val exitCode : Int = ToolRunner.run(new MaxTemperatureDriver, args)
System.exit(exitCode)

}
}

最佳答案

在提交作业时,您需要向 HADOOP_CLASSPATH-libjars 添加任何依赖项,如下例所示:

使用以下命令从(例如)当前目录和 lib 目录添加所有 jar 依赖项:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:`echo *.jar`:`echo lib/*.jar | sed 's/ /:/g'`

请记住,当通过 hadoop jar 启 Action 业时,您还需要通过使用 -libjars 将任何依赖项的 jar 传递给它。我喜欢使用:

hadoop jar <jar> <class> -libjars `echo ./lib/*.jar | sed 's/ /,/g'` [args...]

注意 sed 命令需要不同的分隔符; HADOOP_CLASSPATH: 分隔的,-libjars 需要 , 分隔。

关于scala - Hadoop 作业在 java.lang.ClassNotFoundException 上失败,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/17010741/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com