hadoop - hadoop版本不兼容

Question

我在 Hadoop 0.20.2 版本上开发了一个带有 reduce-side join 的 MapReduce。所有 java 类都可以编译，没有任何错误。我的程序可以正常工作。但是这个版本的 hadoop 在测试虚拟机上。在我的真实集群上，我有另一个版本的 hadoop (hadoop 2.00-cdh4.1.2)。当我想编译 java 类时，它不起作用。我收到如下 100 个错误:

JoinMapper.java:8: package org.apache.hadoop.filecache does not exist
import org.apache.hadoop.filecache.DistributedCache;
                                  ^
JoinMapper.java:9: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
                           ^
JoinMapper.java:10: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
JoinMapper.java:11: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
JoinMapper.java:12: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileInputFormat;
                               ^
JoinMapper.java:13: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileOutputFormat;
                               ^
JoinMapper.java:14: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.InputFormat;
                               ^
JoinMapper.java:15: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobClient;
                               ^
JoinMapper.java:16: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobConf;
                               ^
JoinMapper.java:17: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.MapReduceBase;
                               ^
JoinMapper.java:18: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Mapper;
                               ^
JoinMapper.java:19: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.OutputCollector;
                               ^
JoinMapper.java:20: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Reporter;
                               ^
JoinMapper.java:21: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.TextOutputFormat;
                               ^
JoinMapper.java:22: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;
                                            ^
JoinMapper.java:23: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.Tool;
                             ^
JoinMapper.java:24: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.ToolRunner;
                             ^
JoinMapper.java:26: cannot find symbol
symbol: class MapReduceBase
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Mapper
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                         ^
JoinMapper.java:26: cannot find symbol
symbol: class LongWritable
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                              ^
TextPair.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
TextPair.java:4: cannot find symbol
symbol: class WritableComparable
public class TextPair implements WritableComparable<TextPair> {
                                 ^
TextPair.java:4: interface expected here
public class TextPair implements WritableComparable<TextPair> {
                                                   ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                                              ^
JoinMapper.java:26: interface expected here
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                               ^
NcdcRecordParser.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;   

我知道这个版本没有我需要的包。但我不知道如何解决我的问题。谁能帮帮我？

Answer 1

据我所知，这些 java 包应该仍然存在于最新版本的 Hadoop 中。

我认为您的问题是 hadoop jar 不在您的类路径中。当我尝试在没有它们的情况下编译我的 map reduce 项目时，我遇到了类似的错误。

如果您使用的是 javac，请尝试这样的操作(这使用我的 Ubuntu CDH4 开发机器上的路径位置):

javac -classpath /usr/lib/hadoop/*:/usr/lib/hadoop-0.20-mapreduce/*:/usr/lib/hadoop-hdfs/* JoinMapper.java TextPair.java NcdcRecordParser.java