hadoop - 以编程方式获取总映射并减少hadoop的时间

Question

我正在尝试通过我的MR代码中的所有任务计算映射，改组和归约的总时间。

我需要为每个MapReduce作业检索该信息的帮助。

有人可以发布进行该计算的任何代码段吗？

Answer 1

您需要使用JobClient API，如下所示:
但是，API有一些怪癖。试试看，让我知道我会帮你的。

    JobClient client = null;

         Configuration configuration = new Configuration();

        configuration.set("mapred.job.tracker", jobTrackerURL);
        client = new JobClient(new JobConf(configuration));

        while (true) {


            List<JobStatus> jobEntries = getTrackerEntries(jobName,
                        client);


                    for (JobStatus jobStatus : jobEntries) {


                        JobID jobId = jobStatus.getJobID();
                        String trackerJobName = client.getJob(jobId)
                                .getJobName();
                        TaskReport[] mapReports = client
                                .getMapTaskReports(jobId);
                        TaskReport[] reduceReports = client
                                .getReduceTaskReports(jobId);
                        client.getJob(jobId).getJobStatus().getStartTime();


                        int jobMapper = mapReports.length;
                        mapNumber = +jobMapper;
                        int jobReducers = reduceReports.length;
                        reduceNumber = +jobReducers;


                    }
                }