ruby - hadoop流作业失败无法从SCDynamicStore env加载领域信息:ruby\r:无此类文件或目录

Question

在使用ruby作为我的映射器并减少函数运行hadoop流时，出现以下错误。

packageJobJar: [summarymapper.rb, wcreducer.rb, /var/lib/hadoop/hadoop-unjar6514686449101598265/] [] /var/folders/md/0ww65qrx1_n1nlhrr7hrs8d00000gn/T/streamjob9165241112855689376.jar tmpDir=null
14/06/25 19:54:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/25 19:54:35 WARN snappy.LoadSnappy: Snappy native library not loaded
14/06/25 19:54:35 INFO mapred.FileInputFormat: Total input paths to process : 1
14/06/25 19:54:35 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop/mapred/local]
14/06/25 19:54:35 INFO streaming.StreamJob: Running job: job_201406251944_0005
14/06/25 19:54:35 INFO streaming.StreamJob: To kill this job, run:
14/06/25 19:54:35 INFO streaming.StreamJob: /Users/oladotunopasina/hadoop-1.2.1/libexec/../bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201406251944_0005
14/06/25 19:54:35 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201406251944_0005
14/06/25 19:54:36 INFO streaming.StreamJob:  map 0%  reduce 0%
14/06/25 19:55:18 INFO streaming.StreamJob:  map 100%  reduce 100%
14/06/25 19:55:18 INFO streaming.StreamJob: To kill this job, run:
14/06/25 19:55:18 INFO streaming.StreamJob: /Users/oladotunopasina/hadoop-1.2.1/libexec/../bin/hadoop job  -Dmapred.job.tracker=localhost:8021 -kill job_201406251944_0005
14/06/25 19:55:18 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201406251944_0005
14/06/25 19:55:18 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201406251944_0005_m_000001
14/06/25 19:55:18 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

在检查生成的日志文件时，我看到了

stderr logs

2014-06-25 19:54:38.332 java[8468:1003] Unable to load realm info from SCDynamicStore
env: ruby\r: No such file or directory
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 127
    at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
    at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
    at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
    at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:394)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.Child.main(Child.java:249)

我试过使用此线程 Hadoop environment variables

但仍然没有成功。请帮助。

Answer 1

我通过在Mac上保存.rb文件来解决此问题，似乎我下载的版本已保存为pc文件。 “\ r”是映射器和化简器类中存在的隐藏字符。