c# - MapReduce.SDK : How to wait for MapReduce job?

Question

我正在使用 Microsoft MapReduce SDK 启动仅 Mapper 作业。

调用 hadoop.MapReduceJob.ExecuteJob 立即抛出“响应状态代码不表示成功:404(未找到)”异常。

检查 HDInsight 查询控制台时，作业成功启动并稍后完成。它还会写入正确的输出文件。

我的猜测是，ExecuteJob 试图在作业完成之前访问输出数据。

处理这种情况的正确方法是什么？

using System;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using Microsoft.WindowsAzure.Management.HDInsight;
using Microsoft.Hadoop.MapReduce;
using AzureAnalyzer.MultiAnalyzer;

namespace AzureAnalyzer
{
    class Program
    {
        static void Main(string[] args)
        {
            IHadoop hadoop = Hadoop.Connect(Constants.azureClusterUri, Constants.clusterUser,
            Constants.hadoopUser, Constants.clusterPassword, Constants.storageAccount,
            Constants.storageAccountKey, Constants.container, true);

            try {
                var output = hadoop.MapReduceJob.ExecuteJob<MultiAnalyzerJob>();
            }
            catch (Exception ex) 
            {
                Console.WriteLine("\nException: " + ex.Message);
            }
        }  
    }
}

Answer 1

我有另一种方法来做同样的事情，但需要一些努力，因为它需要将映射器和缩减器文件传输到 hadoop 集群存储。

您需要添加 Microsoft.Hadoop.Client，然后还需要添加 Microsoft Azure HDInsight NuGet 包。

var jobcred = new BasicAuthCredential();
        jobcred.UserName = "clusteruserid";
        jobcred.Password = "clusterpassword";
        jobcred.Server = new Uri("https://clusterurl");

 StreamingMapReduceJobCreateParameters jobpara = new StreamingMapReduceJobCreateParameters()
        {
            JobName="mapreduce",
            Mapper = "Mapper.exe",
            Reducer = "Reducer.exe",
            Input= "wasb:///mydata/input",
            Output = "wasb:///mydata/Output",
            StatusFolder= "wasb:///mydata/sOutput"

        };
        jobpara.Files.Add("wasb:///mydata/Mapper.exe");
        jobpara.Files.Add("wasb:///mydata/Reducer.exe");


 // Create a Hadoop client to connect to HDInsight.
        var jobClient = JobSubmissionClientFactory.Connect(jobcred);


        // Run the MapReduce job.
        JobCreationResults mrJobResults = jobClient.CreateStreamingJob(jobpara);


        // Wait for the job to complete.
        Console.Write("Job running...");
        JobDetails jobInProgress = jobClient.GetJob(mrJobResults.JobId);
        while (jobInProgress.StatusCode != JobStatusCode.Completed
          && jobInProgress.StatusCode != JobStatusCode.Failed)
        {
            Console.Write(".");
            jobInProgress = jobClient.GetJob(jobInProgress.JobId);
            Thread.Sleep(TimeSpan.FromSeconds(10));
        }
        // Job is complete.
        Console.WriteLine("!");
        Console.WriteLine("Job complete!");
        Console.WriteLine("Press a key to end.");
        Console.Read();

希望这对您有所帮助。我能够在不抛出任何异常的情况下运行作业。

这实际上是在等待作业完成。