hadoop - 在Hadoop中， reducer 在随机播放阶段在哪里复制其输入

Question

在Hadoop中，映射器的输出在随机播放阶段复制到reducer。还原器必须从不同的映射器复制其对应的分区。 reducer 在开始实际的减速过程之前将输入存储在哪里？

Answer 1

The map outputs are copied to the reduce task JVM’s memory if they are small enough (the buffer’s size is controlled by mapred.job.shuffle.input.buffer.percent, which specifies the proportion of the heap to use for this purpose); otherwise, they are copied to disk. When the in-memory buffer reaches a threshold size (controlled by mapred.job.shuffle.merge.percent) or reaches a threshold number of map outputs (mapred.inmem.merge.threshold), it is merged and spilled to disk. If a combiner is specified, it will be run during the merge to reduce the amount of data written to disk.

引用-Hadoop权威指南