csv - 从CSV文件创建Hive表时的唯一ID

Question

我有一个要导出为Hive表的CSV文件列表，但是我很确定某些记录在CSV中是多余的。 CSV中的每个记录/行都由一个键标识，我想使用该键作为主键来生成表。我将如何生成Hive表，以确保没有重复的行？

Answer 1

ROW_NUMBER() OVER([partition_by_clause] order_by_clause)

返回以1开头的升序整数序列。

select x, row_number() over(order by x, property) as row_number, property from int_t;
+----+------------+----------+
| x  | row_number | property |
+----+------------+----------+
| 1  | 1          | odd      |
| 1  | 2          | square   |
| 2  | 3          | even     |
| 2  | 4          | prime    |
| 3  | 5          | odd      |
| 3  | 6          | prime    |
| 4  | 7          | even     |
| 4  | 8          | square   |
| 5  | 9          | odd      |
| 5  | 10         | prime    |
| 6  | 11         | even     |
| 6  | 12         | perfect  |
| 7  | 13         | lucky    |
| 7  | 14         | lucky    |
| 7  | 15         | lucky    |
| 7  | 16         | odd      |
| 7  | 17         | prime    |
| 8  | 18         | even     |
| 9  | 19         | odd      |
| 9  | 20         | square   |
| 10 | 21         | even     |
| 10 | 22         | round    |
+----+------------+----------+