java - WordCount Mapreduce 程序中的 IO 异常 "System cannot find the file specified"

Question

我对 Map/Reduce 和 Hadoop 相当陌生。我正在尝试编写一个 wordccount mapreduce 程序，并使用 eclipse 在本地运行它。我已经指定了输入文件的路径以及输出目录。当我编译程序时，它抛出 IO 异常“系统找不到指定的文件。”

我的代码如下所示

import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobClient;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.Job;

public class WordCount {
public static class Wordcountmapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable>{
    private final static IntWritable one = new IntWritable(1);
    private Text word=new Text();
    @Override
    public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output,
            Reporter reporter) throws IOException {

        String line = value.toString();
        System.out.println("Line " + line);
        StringTokenizer token = new StringTokenizer(line,",");

        while(token.hasMoreTokens())

        {
            word.set(token.nextToken());
            output.collect(word, one);
            System.out.println("hii  " + word + " " + one);
        }   
    }   
}
public static class wordcountreducer extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterator<IntWritable> value,
            OutputCollector<Text, IntWritable> output, Reporter reporter)
            throws IOException {
        int sum = 0 ;
        System.out.println("Inside Reducer" + value.hasNext());
        System.out.println("Key = " + key );
        while(value.hasNext()) {
            sum += value.next().get();
        }
        output.collect(key,new IntWritable(sum));
        System.out.println(key + " " + sum);
    }
}
public static void main(String[] args) throws IOException {
    JobConf conf = new JobConf(WordCount.class);
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(IntWritable.class);
    conf.setMapperClass(Wordcountmapper.class);
    conf.setReducerClass(wordcountreducer.class);
    FileInputFormat.addInputPath(conf, new Path("C:\\WordCount\\wordcount.txt"));
    String outputfile = "C:\\WordCount\\Output\\a.txt";
    FileOutputFormat.setOutputPath(conf, new Path(outputfile));
    JobClient.runJob(conf);
}
}

我在这里遗漏了什么吗？我正在使用安装了 hadoop 插件的 Eclipse Juno 来运行 map 缩减程序。

抛出的错误如下

15/01/22 17:54:43 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15/01/22 17:54:43 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/01/22 17:54:43 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/01/22 17:54:43 INFO Configuration.deprecation: mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename
15/01/22 17:54:43 ERROR security.UserGroupInformation: PriviledgedActionException as:Soorya S (auth:SIMPLE) cause:java.io.IOException: Cannot run program "chmod": CreateProcess error=2, The system cannot find the file specified.
Exception in thread "main" java.io.IOException: Cannot run program "chmod": CreateProcess error=2, The system cannot find the file specified.
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:471)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:384)
    at org.apache.hadoop.util.Shell.run(Shell.java:359)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:569)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:658)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:641)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:435)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:277)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:122)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:969)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:963)
    at java.security.AccessController.doPrivileged(AccessController.java:284)
    at javax.security.auth.Subject.doAs(Subject.java:573)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:963)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:937)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1375)
    at com.ibm.hadoop.WordCount.main(WordCount.java:63)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
    at java.lang.reflect.Method.invoke(Method.java:611)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified.
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:92)
    at java.lang.ProcessImpl.start(ProcessImpl.java:41)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:464)
    ... 23 more
getting access token
  [getToken] got user access token

getting primary group
  [getPrimaryGroup] Got TokenPrimaryGroup info
  [getPrimaryGroup] primaryGroup: S-1-5-21-184784153-2138975554-913327727-513
getting supplementary groups
  [getGroups] Got TokenGroups info
  [getGroups] group 0: S-1-5-21-184784153-2138975554-913327727-513
  [getGroups] group 1: S-1-1-0
  [getGroups] group 2: S-1-5-114
  [getGroups] group 3: S-1-5-32-544
  [getGroups] group 4: S-1-5-32-545
  [getGroups] group 5: S-1-5-4
  [getGroups] group 6: S-1-2-1
  [getGroups] group 7: S-1-5-11
  [getGroups] group 8: S-1-5-15
  [getGroups] group 9: S-1-5-113
  [getGroups] group 10: S-1-5-5-0-576278
  [getGroups] group 11: S-1-2-0
  [getGroups] group 12: S-1-5-64-10
  [getGroups] group 13:S-1-16-8192

Answer 1

我有同样的问题(我假设你使用的是Windows)。发生这种情况是因为需要设置指向“chmod”可执行文件的系统路径变量。您可以按照此处描述的步骤 running hadoop on window-7 64 bit