Hadoop:如何使用上下文对象在减少步骤中找出partition_Id

Question

在 Hadoop API 版本中。 0.20 及更高版本引入了 Context 对象，而不是 JobConf。

我需要使用 Context 对象找出:

1. 当前 Reducer 的 partition_id

2. 输出文件夹

使用过时的 JobConf，我可以通过以下方式找到当前 Reducer 的 partition_id:

public void configure(JobConf conf){
  int  current_partition = conf.getInt("mapred.task.partition",-1);
}

我认为我需要在方法内部使用 Context 对象

public void setup(Context c)

但是怎么办？输出文件夹名称呢？

Answer 1

如果要获取分区，可以使用context.getTaskAttemptID().getTaskID().getId()。任务 ID 由分区 ID 创建。我把相关代码列在这里，大家可以查看ReduceTaskImpl的代码, TaskImpl和 MRBuilderUtils自己。

public TaskImpl(JobId jobId, TaskType taskType, int partition,
    EventHandler eventHandler, Path remoteJobConfFile, JobConf conf,
    TaskAttemptListener taskAttemptListener, OutputCommitter committer,
    Token jobToken,
    Credentials credentials, Clock clock,
    Map completedTasksFromPreviousRun, int startCount,
    MRAppMetrics metrics, AppContext appContext) {
  this.conf = conf;
  this.clock = clock;
  this.jobFile = remoteJobConfFile;
  ReadWriteLock readWriteLock = new ReentrantReadWriteLock();
  readLock = readWriteLock.readLock();
  writeLock = readWriteLock.writeLock();
  this.attempts = Collections.emptyMap();
  maxAttempts = getMaxAttempts();
  taskId = MRBuilderUtils.newTaskId(jobId, partition, taskType);
  this.partition = partition;
  ...
}

public static TaskId newTaskId(JobId jobId, int id, TaskType taskType) {
  TaskId taskId = Records.newRecord(TaskId.class);
  taskId.setJobId(jobId);
  taskId.setId(id);
  taskId.setTaskType(taskType);
  return taskId;
}