hadoop - Hive 总是给出 "Number of reduce tasks determined at compile time: 1"，无论我做什么

Question

create external table if not exists my_table
(customer_id STRING,ip_id STRING)
location 'ip_b_class';

然后:

hive> set mapred.reduce.tasks=50;
hive> select count(distinct customer_id) from my_table;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1

里面有160GB，1个reducer需要很长时间...

[ihadanny@lvshdc2en0011 ~]$ hdu 
Found 8 items
162808042208   hdfs://horton/ip_b_class

...

Answer 1

从逻辑上讲，这里不能有多个 reducer。除非来自各个 map task 的所有不同客户 ID 都集中在一个地方，否则无法建立区别性并且无法生成单一计数。换句话说，除非您将所有客户 ID 堆放在一个地方，否则您不能说每个客户 ID 都是不同的并最终计算它们。