python - hadoop-streaming: “No room for reduce task.”减少处于挂起状态的任务

Question

我的 map task 成功完成，我可以看到应用程序日志，但reducer保持挂起状态

Kind    % Complete  Num Tasks   Pending Running Complete    Killed  Failed/Killed Task Attempts
map        100.00%  200 0   0   200 0   0 / 40
reduce  0.00%   1   1   0   0   0   0 / 0  

当我看 reduce task 时，我看到

All Task Attempts

No Task Attempts found

当我看到hadoop-hduser-jobtracker-master.log时，看到以下内容:

2011-10-31 00:00:00,238 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67730714624 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:01,634 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68041449472 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:03,863 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397398528 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,532 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67729956864 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,695 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67998732288 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:06,865 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:08,057 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67948941312 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:09,566 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67728035840 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:11,118 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67907883008 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:12,258 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:14,119 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67867385856 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:16,061 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:17,120 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67813683200 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:19,386 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67726835712 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:20,122 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67896156160 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:21,949 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:23,124 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68150616064 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,078 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,762 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67725717504 bytes free; but we expect reduce input to take 184825113602

如何解决此问题？

请帮忙

Answer 1

运行 reducer 的主机上的磁盘太小。您的 reducer 输入大约需要。 172 GB，但您只有63 GB的可用空间。

也许合并器可能对您有用。在这里看看:http://tumblr.mobocracy.net/post/6589143971/using-a-combiner-with-hadoop-jobs