hadoop - 计算列上的 Hive 分区修剪

Question

我在 Hive 上有几个表，我的查询试图检索过去 x 天的数据。当我使用直接日期时，Hive 正在修剪分区，但当我改用公式时，Hive 正在执行全表扫描。

select *
from   f_event
where  date_key > 20160101;

scanned partitions..

s3://...key=20160102 [f]
s3://...key=20160103 [f]
s3://...key=20160104 [f]

比方说，如果我使用公式来获取过去 4 周的数据

Select count(*)
From    f_event f
Where  date_key  > from_unixtime(unix_timestamp()-2*7*60*60*24, 'yyyyMMdd')

这是扫描表中的所有分区。

环境:Hadoop 2.6.0、EMR、Hive on S3、Hive 1.0.0

Answer 1

当过滤表达式包含非确定性函数(例如 unix_timestamp())时，Hive 不会触发分区修剪。

the discussion 中提到了一个很好的理由:

Imagine a situation where you had:

WHERE partition_column = f(unix_timestamp()) AND ordinary_column =
f(unix_timestamp).

The right hand side of the predicate has to be evaluated at map-time, whereas you're assuming that left hand side should be evaluated at compile time, which means you have two different values of unix_timestamp() floating around, which can only end badly.