c# - .NET 中的 Hadoop 流式处理

Question

我在伪分布式模式下运行 hadoop，并使用 hadoop 流来执行我的 map-reduce 操作。但问题是我不断收到 Streaming Job Failed 错误消息。以下是日志:

标准错误日志

java.io.IOException: Cannot run program "input/StdInOut.exe": CreateProcess error=2, The system cannot find the file specified
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:166)
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(ProcessImpl.java:81)
at java.lang.ProcessImpl.start(ProcessImpl.java:30)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 20 more

我知道它说 java.io.IOException: Cannot run program "input/StdInOut.exe": CreateProcess error=2, The system cannot find the file specified 但文件同时存在于本地和同一路径上的 hdfs。

这是我自己的映射器的实现:

static void Main(string[] args)
{
  string s;
  while ((s = Console.ReadLine()) != null) 
  {
    string[] words = s.Split(' ');
    foreach (var word in words) 
    {
      //Setting occurance of each word to 1
      Console.WriteLine(word + "\t" + 1);
    }
  }
}

我正在使用这个命令在 ssh 中执行我的工作:

bin/hadoop jar contrib/streaming/hadoop-*-streaming.jar -input input/sample.txt -output output -mapper input/StdInOut.exe -reducer NONE

有什么解决办法吗？

Answer 1

错误实际上是映射器的相对路径。