java - 在 org.apache.hadoop.util.ReflectionUtils.setJobConf 配置对象时出错

Question

关于此错误，我遇到了很多问题，但找不到任何可以解决我的问题的解决方案。在这里，我正在使用 Hadoop 对 Twitter 数据实现情绪分析。

主类:

public class SentimentAnalysis extends Configured implements Tool{
private static File file;

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
    Classify classify = new Classify();

    /**
     * Mapper which reads Tweets text file Store 
     * as <"Positive",1> or <"Negative",1>
     */
    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        String line = value.toString();//streaming each tweet from the text file
        if (line != null) {
            word.set(classify.classify(line)); //invoke classify class to get tweet group of each text
            output.collect(word, one);
        } else {
            word.set("Error");
            output.collect(word, one);//Key,value for Mapper
        }
    }
}
public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    /**
     * Count the frequency of each classified text group
     */

    @Override
    public void reduce(Text key, Iterator<IntWritable> classifiedText,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0;
        while (classifiedText.hasNext()) {
            sum += classifiedText.next().get(); //Sum the frequency
        }
        output.collect(key, new IntWritable(sum));
    }
}
public static class Classify {
    String[] categories;
    @SuppressWarnings("rawtypes")
    LMClassifier lmc;

    /**
     * Constructor loading serialized object created by Model class to local
     * LMClassifer of this class
     */
    @SuppressWarnings("rawtypes")
    public Classify() {
        try {

            lmc = (LMClassifier) AbstractExternalizable.readObject(file);
            categories = lmc.categories();
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * Classify whether the text is positive or negative based on Model object
     * 
     * @param text
     * @return classified group i.e either positive or negative
     */
    public String classify(String text) {
        ConditionalClassification classification = lmc.classify(text);
        return classification.bestCategory();
    }
}

public static void main(String[] args) throws Exception {
    int ret = ToolRunner.run(new SentimentAnalysis(), args);
    System.exit(ret);
}

@Override
public int run(String[] args) throws Exception {
    if(args.length < 2) {
        System.out.println("Invalid input and output directories");
        return -1;
    }
    JobConf conf = new JobConf(getConf(), SentimentAnalysis.class);
    conf.setJobName("sentimentanalysis");
    conf.setJarByClass(SentimentAnalysis.class);
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);
    conf.setMapOutputKeyClass(Text.class);
    conf.setMapOutputValueClass(IntWritable.class);
    conf.setMapperClass(Map.class);
    //conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);
    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputFormat(TextOutputFormat.class);
    FileInputFormat.setInputPaths(conf, new Path(args[0]));
    FileOutputFormat.setOutputPath(conf, new Path(args[1]));
    file = new File(args[2]);
    JobClient.runJob(conf);
    return 0;
}
}

错误:

[cloudera@localhost ~]$ hadoop jar Sentiment.jar SentimentAnalysis test.txt SentimentOutput classifier.txt

Test.txt 包含少量需要分析其情绪的推文。 Classifier.txt 是一个编码文本文件，可帮助分类 (LMClassifier) 类分析 Test.txt 中存在的推文。

14/10/05 20:59:23 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/10/05 20:59:24 INFO mapred.FileInputFormat: Total input paths to process : 1
14/10/05 20:59:24 INFO mapred.JobClient: Running job: job_201410041909_0035
14/10/05 20:59:25 INFO mapred.JobClient:  map 0% reduce 0%
14/10/05 20:59:41 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:41 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:52 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 20:59:53 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_1, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:04 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000000_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:04 INFO mapred.JobClient: Task Id : attempt_201410041909_0035_m_000001_2, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:413)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
14/10/05 21:00:19 INFO mapred.JobClient: Job complete: job_201410041909_0035
14/10/05 21:00:19 INFO mapred.JobClient: Counters: 7
14/10/05 21:00:19 INFO mapred.JobClient:   Job Counters 
14/10/05 21:00:19 INFO mapred.JobClient:     Failed map tasks=1
14/10/05 21:00:19 INFO mapred.JobClient:     Launched map tasks=8
14/10/05 21:00:19 INFO mapred.JobClient:     Data-local map tasks=8
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=98236
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
14/10/05 21:00:19 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1416)
at SentimentAnalysis.run(SentimentAnalysis.java:124)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at SentimentAnalysis.main(SentimentAnalysis.java:101)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Answer 1

我个人没有使用 Hadoop 的经验，但是如果您“查看”堆栈跟踪，它似乎是 org.apache.hadoop.util.ReflectionUtils.setJobConf 中的运行时异常...

private static void setJobConf(Object theObject, Configuration conf) {
75     //If JobConf and JobConfigurable are in classpath, AND
76     //theObject is of type JobConfigurable AND
77     //conf is of type JobConf then
78     //invoke configure on theObject
79     try {
80       Class<?> jobConfClass = 
81         conf.getClassByName("org.apache.hadoop.mapred.JobConf");
82       Class<?> jobConfigurableClass = 
83         conf.getClassByName("org.apache.hadoop.mapred.JobConfigurable");
84        if (jobConfClass.isAssignableFrom(conf.getClass()) &&
85             jobConfigurableClass.isAssignableFrom(theObject.getClass())) {
86         Method configureMethod = 
87           jobConfigurableClass.getMethod("configure", jobConfClass);
88         configureMethod.invoke(theObject, conf);
89       }
90     } catch (ClassNotFoundException e) {
91       //JobConf/JobConfigurable not in classpath. no need to configure
92     } catch (Exception e) {
93       throw new RuntimeException("Error in configuring object", e);
94     }
95   }

显然，JobConf 和 JobConfigurable 类都在类路径中(否则它会通过 CNFE catch block )所以发生了另一个异常......看起来好像嵌套异常是 java.lang.reflect.InvocationTargetException建议上面第 88 行的“调用”有问题。

因此，尝试使用传入的配置在目标作业实例上调用“configure”方法是失败的。

InvocationTargetException 可能包装了实际的因果异常，因此您需要以某种方式在顶层捕获 RuntimeException，然后使用 e.getCause().getCause().printStackTrace() 找出配置方法调用失败的原因。