java - Hadoop 类未找到异常

Question

我正在 hadoop 上开发简单的程序，我遵循了本教程的步骤: http://www.bogotobogo.com/Hadoop/BigData_hadoop_Creating_Java_Wordcount_Project_with_Eclipse_MapReduce2.php

即使我在两台不同的机器上试过，它仍然显示这个异常:

Exception in thread "main" java.lang.ClassNotFoundException: test.java
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

---

package pa2;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;


public class test extends Configured implements Tool{


public int run(String[] args) throws Exception
{ if (args.length<2)
{
    System.out.println("plz give proper arguments");
    return -1;
}
      //creating a JobConf object and assigning a job name for identification purposes
      JobConf conf = new JobConf(test.class);

      FileInputFormat.setInputPaths(conf, new Path(args[0]));
      FileOutputFormat.setOutputPath(conf, new Path(args[1]));

      conf.setMapperClass(mapper.class);

      conf.setMapOutputKeyClass(Text.class);
      conf.setMapOutputValueClass(IntWritable.class);

      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(IntWritable.class);

      JobClient.runJob(conf);

      return 0;
}


public static void main(String[] args) throws Exception
{
      // this main function will call run method defined above.
  int exitcode = ToolRunner.run(new test(),args);
      System.exit(exitcode);
}
}

你能告诉我这里有什么问题吗？

更新:

映射器类:

package pa2;
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;


public class mapper extends MapReduceBase 
        implements Mapper<LongWritable,Text, Text, IntWritable>
{
            public void map(LongWritable Key, Text value,
            OutputCollector<Text, IntWritable> output, Reporter r)
            throws IOException {


            int i=0;
            String [] array = new String [50];


                        String name;
                        String year;
                        String s=value.toString();

                        for (String word:s.split(",")){

                   word = s.substring(0, s.indexOf(",")+1);
                   year= word.substring(0, s.indexOf(",")+1);
                   name=word.substring(s.indexOf(",")+1);
                   int theyear= Integer.parseInt(year);


                   if(theyear<2000){
                        array[i] =name;
                        output.collect(new Text(word),  new IntWritable(1));

                        i++;}

                    }       
    }
}

我还没有写 reducer 类。我将项目导出为 jar 文件，并制作了一个名为 movies 的文本文件作为程序的输入。然后在终端中写下:

[cloudera@quickstart ~]$ cd workspace
[cloudera@quickstart workspace]$ ls
pa2  pa2.jar  training
[cloudera@quickstart workspace]$ hadoop jar pa2.jar test movies.txt output.txt
Exception in thread "main" java.lang.ClassNotFoundException: test
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Answer 1

不能保证这是解决当前问题的方法，但是

package pa2;

这是附加到类名的。换句话说，完全限定的类名是 pa2.test。

所以，试试

hadoop jar ~/workspace/pa2.jar pa2.test input output

如果您使用该教程中显示的默认包，则无需在命令行中指定包。