java - 在 hadoop 中运行 jar 文件时出错

Question

在 hadoop 中运行 jar 文件时，出现空指针异常。我无法理解问题所在。

以下是我的驱动类:

package mapreduce;

import java.io.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;


public class StockDriver extends Configured implements Tool
{
      public int run(String[] args) throws Exception
      {
            //creating a JobConf object and assigning a job name for identification purposes
            JobConf conf = new JobConf(getConf(), StockDriver.class);
            conf.setJobName("StockDriver");

            //Setting configuration object with the Data Type of output Key and Value
            conf.setOutputKeyClass(Text.class);
            conf.setOutputValueClass(IntWritable.class);

            //Providing the mapper and reducer class names
            conf.setMapperClass(StockMapper.class);
            conf.setReducerClass(StockReducer.class);

            File in = new File(args[0]);
            int number_of_companies = in.listFiles().length;
            for(int iter=1;iter<=number_of_companies;iter++)
            {
                Path inp = new Path(args[0]+"/i"+Integer.toString(iter)+".txt");
                Path out = new Path(args[1]+Integer.toString(iter));
                //the HDFS input and output directory to be fetched from the command line
                FileInputFormat.addInputPath(conf, inp);
                FileOutputFormat.setOutputPath(conf, out);
                JobClient.runJob(conf);
            }
            return 0;
      }

      public static void main(String[] args) throws Exception
      {
            int res = ToolRunner.run(new Configuration(), new StockDriver(),args);
            System.exit(res);
      }
}

映射器类:

package mapreduce;

import java.io.IOException;
import gonn.ConstraintTree;

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;

public class StockMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>
{
      //hadoop supported data types
      private static IntWritable send;
      private Text word;

      //map method that performs the tokenizer job and framing the initial key value pairs
      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException
      {
            //taking one line at a time and tokenizing the same
            String line = value.toString();
            String[] words = line.split(" ");
            String out = ConstraintTree.isMain(words[1]);
            word = new Text(out);

            send = new IntWritable(Integer.parseInt(words[0]));
            output.collect(word, send);
       }
}

reducer 类:

package mapreduce;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;

public class StockReducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable>
{
      //reduce method accepts the Key Value pairs from mappers, do the aggregation based on keys and produce the final output
      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException
      {
            int val = 0;

            while (values.hasNext())
            {
               val += values.next().get();
            }
            output.collect(key, new IntWritable(val));
      }
}

堆栈跟踪:

Exception in thread "main" java.lang.NullPointerException
    at mapreduce.StockDriver.run(StockDriver.java:29)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at mapreduce.StockDriver.main(StockDriver.java:44)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

当我尝试使用 java -jar myfile.jar args... 运行 jar 文件时，它工作正常。但是，当我尝试使用 hadoop jar myfile.jar [MainClass] args... 在 hadoop 集群上运行它时，出现了错误。

澄清一下，第 29 行是 int number_of_companies = in.listFiles().length;

Answer 1

问题的原因是使用File api读取HDFS文件。如果您使用不存在的路径创建 File 对象，则 listFiles 方法返回 null。作为 HDFS 中的输入目录(我假设)，本地文件系统不存在它，NPE 来自:

in.listFiles().length

使用以下命令提取 HDFS 目录中的文件数:

FileSystem fs = FileSystem.get(new Configuration());
int number_of_companies = fs.listStatus(new Path(arg[0])).length;