google-bigquery - 将数据从 Redshift 迁移到 BigQuery

Question

我想将分布在大约 100 个表中的 1 TB 数据从 Redshift 迁移到 BigQuery。

是否有用于此数据迁移的工具？如果像“pgloader”(用于从 MySQL 迁移到 PostgreSQL)这样的东西可用，它会节省很多时间吗？

Answer 1

2019 年更新 - 现在 officially supported :

https://cloud.google.com/bigquery/docs/redshift-migration

我知道在 GitHub 上有 2 个开源替代方案:

https://github.com/iconara/bigshift

https://github.com/uswitch/bqshift

两者似乎都维护得很好，而 bigshift 已经存在了更长的时间并显示了更完整的文档(目前)。

引用 bigshift 文档，说明为什么该工具在进行此迁移时非常有用:

The CSV produced by Redshift's UNLOAD can't be loaded into BigQuery no matter what options you specify on either end. Redshift can quote all fields or none, but BigQuery doesn't allow non-string fields to be quoted. The format of booleans and timestamps are not compatible, and they expect quotes in quoted fields to be escaped differently, to name a few things.

This means that a lot of what BigShift does is make sure that the data that is dumped from Redshift is compatible with BigQuery. To do this it reads the table schema and translates the different datatypes while the data is dumped. Quotes are escaped, timestamps formatted, and so on.