java - Windows上的Hadoop:获取异常 “not a valid DFS filename”

Question

在起步阶段，我是初次接触Hadoop和挣扎的人。
在eclipse中，我编写了单词计数程序，并为wordcount程序创建了JAR。

我正在尝试使用以下hadoop命令运行它:

$ ./hadoop jar C:/cygwin64/home/PAKU/hadoop-1.2.1/wordcount.jar com.hadoopexpert.WordCountDriver file:///C:/cygwin64/home/PAKU/work/hadoopdata/tmp/dfs/ddata/file.txt file:///C:/cygwin64/home/PAKU/hadoop-dir/datadir/tmp/output

我得到像这样的异常:

Exception in thread "main" java.lang.IllegalArgumentException: Pathname /C:/cygwin64/home/PAKU/work/hadoopdata/tmp/mapred/staging/PAKU/.staging from hdfs://localhost:50000/C:/cygwin64/home/PAKU/work/hadoopdata/tmp/mapred/staging/PAKU/.staging is not a valid DFS filename.
        at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:143)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:554)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:788)
        at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:109)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:942)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:550)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:580)
        at com.hadoopexpert.WordCountDriver.main(WordCountDriver.java:30)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

注意:我正在使用cygwin在Windows上运行hadoop。

码:

public class WordCountDriver {
    public static void main(String[] args) {
        try {
            Job job = new Job();
            job.setMapperClass(WordCountMapper.class);
            job.setReducerClass(WordCountReducer.class);
            job.setMapOutputKeyClass(Text.class);
            job.setMapOutputValueClass(IntWritable.class);

            job.setOutputKeyClass(Text.class);
            job.setOutputValueClass(IntWritable.class);

            job.setJarByClass(WordCountDriver.class);

            FileInputFormat.setInputPaths(job, new Path(args[0]));
            FileOutputFormat.setOutputPath(job, new Path(args[1]));
            try {
                System.exit(job.waitForCompletion(true) ? 0 :-1);
            } catch (ClassNotFoundException e) {
                e.printStackTrace();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            } catch (IOException e) {
            e.printStackTrace();
        }
    }
}


public class WordCountReducer extends Reducer<Text,IntWritable, Text, IntWritable>{
    public void reduce(Text key, Iterable<IntWritable> value, Context context){
        int total = 0;
        while(value.iterator().hasNext()){
            IntWritable i = value.iterator().next();
            int i1= i.get();
            total += i1;
        }
        try {
            context.write(key, new IntWritable(total));
        } catch (IOException e) {
            e.printStackTrace();
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }
}


public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
    public void map(LongWritable key, Text value, Context context){
        String s = value.toString();
        for(String word :s.split(" ")){
            Text text = new Text(word);
            IntWritable intW = new IntWritable(1);
            try {
                context.write(text, intW);
            } catch (IOException e) {
                e.printStackTrace();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }  
}

谁能帮助我运行我的第一个hadoop程序。

提前致谢。

Answer 1

您已经为FileInputFormat和FileOutputFormat指定了本地路径。

将文件放入hdfs，然后使用hdfs路径。

脚步:

首先将put(或copyFromLocal)文件添加到hdfs:

hdfs dfs -put /local/file/locaion hdfs://ip_add:port/hdfs_location

您可以使用ls检查文件:

hdfs dfs -ls /hdfs_location/

现在将hdfs位置作为输入参数，并为输出提供新目录。