hadoop - 什么时候在 mapreduce 作业执行中分配 reducer 的数量/节点？

Question

在阅读 MapReduce 时，我读到以下有趣的行:

"But how do the Reducer’s know which nodes to query to get their partitions? This happens through the Application Master. As each Mapper instance completes, it notifies the Application Master about the partitions it produced during its run. Each Reducer periodically queries the Application Master for Mapper hosts until it has received a final list of nodes hosting its partitions."

我在这里有一个疑问。当他们说 Each Reducer 时，它到底是什么意思？ reducer 是否会在 map 阶段开始之前分配，以及 reducer 节点是如何选择的？

Answer 1

Reducer 可以在映射完成数据处理之前启动。一旦他们开始，他们就可以从映射器机器中提取数据，但只有在所有映射器完成数据处理后，他们才会开始处理。

ma​​pred.reduce.slowstart.completed.maps 是配置此行为的属性。更多特性信息here .