azure - pipelinemapred waitoutputthreads 子进程失败，代码为 255

Question

我正在 Azure HDInsight (Hadoop) 上运行一个简单的 Map/Reduce 作业。映射器失败并出现以下错误(来自 JobTracker 日志)...

2014-03-27 16:42:50,330 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201403262112_0004_m_000000_0: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 255
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1233)
at org.apache.hadoop.mapred.Child.main(Child.java:260)

我正在使用 Hadoop.Connect 方法连接到集群...

// Connect to the HDInsight cluster
var hadoop = Hadoop.Connect(hadoopClusterUri,
                            hadoopUserName,
                            localHadoopUser,
                            hadoopPassword,
                            asvAccount,
                            asvKey,
                            asvContainer,
                            createContinerIfNotExist);

由于作业已启动并记录，我假设连接成功。作为一个简单的测试，我发出少量数据......

context.EmitKeyValue("This is a key", "This is a value");

完全被难住了，比尔

更新这个 MapReduce 产生了两个作业。第一个成功完成，第二个失败，但出现以下异常...

Unhandled Exception: Microsoft.Hadoop.MapReduce.StreamingException: The user type could not be loaded. DLL=MapReduce.exe, Type=PrescienceRx.MapReduce.BeneficiaryMapper ---> System.BadImageFormatException: Could not load file or assembly 'file:///c:\apps\temp\hdfs\mapred\local\taskTracker\admin\jobcache\job_201403262112_0006\attempt_201403262112_0006_m_000001_0\work\MapReduce.exe' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection, Boolean suppressSecurityChecks)
at System.Reflection.RuntimeAssembly.InternalLoadFrom(String assemblyFile, Evidence securityEvidence, Byte[] hashValue, AssemblyHashAlgorithm hashAlgorithm, Boolean forIntrospection, Boolean suppressSecurityChecks, StackCrawlMark& stackMark)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at Microsoft.Hadoop.MapReduce.MapperMain.Main()
--- End of inner exception stack trace ---

在 Microsoft.Hadoop.MapReduce.MapperMain.Main() 在 Microsoft.HdInsight.MapDriver.Program.Main(String[] args

Answer 1

看起来System.BadImageFormatException:无法加载文件或程序集可能是因为加载程序的.Net版本与程序集之间不匹配。尝试将其设置回 4.0，然后在 NuGet 包管理器窗口中执行 update-package -reinstall Microsoft.Hadoop.MapReduce 以确保您拥有所有正确的版本。