hadoop - 通过 Web 界面跟踪 Hadoop 作业状态？ (将Hadoop暴露给公司内部客户)

Question

我想开发一个网站，允许公司内的分析师运行 Hadoop 作业(从一组定义的作业中选择)并查看其作业的状态\进度。

有没有一种简单的方法可以通过 Ruby\Python 执行此操作(获取正在运行的作业状态等)？您如何向公司内部客户公开 Hadoop 集群？

Answer 1

我找到了一种在 JobTracker 上获取职位信息的方法。这是代码:

    Configuration conf = new Configuration();
    conf.set("mapred.job.tracker", "URL");

    JobClient client = new JobClient(new JobConf(conf));

    JobStatus[] jobStatuses = client.getAllJobs();
    for (JobStatus jobStatus : jobStatuses) {

        long lastTaskEndTime = 0L;

        TaskReport[] mapReports = client.getMapTaskReports(jobStatus.getJobID());
        for (TaskReport r : mapReports) {
            if (lastTaskEndTime < r.getFinishTime()) {
                lastTaskEndTime = r.getFinishTime();
            }
        }

        TaskReport[] reduceReports = client.getReduceTaskReports(jobStatus.getJobID());
        for (TaskReport r : reduceReports) {
            if (lastTaskEndTime < r.getFinishTime()) {
                lastTaskEndTime = r.getFinishTime();
            }
        }
        client.getSetupTaskReports(jobStatus.getJobID());
        client.getCleanupTaskReports(jobStatus.getJobID());

        System.out.println("JobID: " + jobStatus.getJobID().toString() + 
                            ", username: " + jobStatus.getUsername() + 
                            ", startTime: " + jobStatus.getStartTime() + 
                            ", endTime: " + lastTaskEndTime + 
                            ", Durration: " + (lastTaskEndTime - jobStatus.getStartTime()));

    }