hadoop - Hadoop 完成的作业和退休的作业之间的区别

Question

标题使问题非常清楚。为什么在 jobtracker 中有两个不同的部分用于完成和退休的工作？

谢谢。

Answer 1

作业退役是 jobtracker 将作业持久保存到磁盘并清理内存的正常过程。您可以在 the Cloudera blog 上阅读更多详细信息:

Once a job is complete it is kept in memory (up to mapred.jobtracker.completeuserjobs.maximum) and on disk as per the above. There is a configuration value that controls the overall retirement policy of completed jobs:

Key: mapred.jobtracker.retirejob.interval
Default: 24 * 60 * 60 * 1000 (1 day)
In other words, completed jobs are retired after one day by default. The check for jobs to be retired is done by default every minute and can be controlled with:

Key: mapred.jobtracker.retirejob.check
Default: 60 * 1000 (60s in msecs)
The check runs continually while the JobTracker is running. If a job is retired it is simply removed from the in-memory list of the JobTracker (it also removes all Tasks for the job etc.). Jobs are not retired under at least 1 minute (hardcoded in JobTracker.java) of their finish time. The retire call also removes the JobTracker Local (see above) file for the job. All that is left are the two files per retired job in the history directory (hadoop.job.history.location) plus – if enabled – the Per Job files (hadoop.job.history.user.location).