amazon-web-services - AWS 更新雅典娜元 : Glue Crawler vs MSCK Repair Table

Question

当新分区添加到 Athena 表时，我们可以使用 Glue Crawler 或 MSCK REPAIR TABLE 来更新元信息。他们的成本是多少？哪一个是首选？

Answer 1

MSCK REPAIR TABLE命令要求您的 S3 key 将分区方案包含为 documented here .如果您的 S3 key 不包含分区方案，则 MSCK REPAIR TABLE命令将返回丢失的分区，但您仍然需要添加它们。另外一个区别是 MSCK REPAIR TABLE命令可以在 30 分钟后超时(默认 Athena 查询时间长度)，而胶水爬​​虫不会。

以下是定价信息:

Glue Crawler :

There is an hourly rate for AWS Glue crawler runtime to discover data and populate the AWS Glue Data Catalog. You are charged an hourly rate based on the number of Data Processing Units (or DPUs) used to run your crawler. A single Data Processing Unit (DPU) provides 4 vCPU and 16 GB of memory. You are billed in increments of 1 second, rounded up to the nearest second, with a 10-minute minimum duration for each crawl. Use of AWS Glue crawlers is optional, and you can populate the AWS Glue Data Catalog directly through the API.

Pricing

For all AWS Regions where AWS Glue is available: $0.44 per DPU-Hour, billed per second, with a 10-minute minimum per crawler run

Athena :

There are no charges for Data Definition Language (DDL) statements like CREATE/ALTER/DROP TABLE, statements for managing partitions, or failed queries.

但是，除了这两个命令之外，您仍然会产生 S3 成本。引用: AWS Athena: does `msck repair table` incur costs?

我的意见是，如果可以，最好在添加新数据后自己管理分区。

'ALTER TABLE database.table ADD
PARTITION (partition_name='PartitionValue') location 's3://bucket/path/partition'

如果被迫使用 Glue 或 Athena，我会评估哪种方式更适合您的流程。 MSCK REPAIR TABLE命令可能更易于管理，但如果分区中有大量数据或未正确分区，则可能会遇到麻烦。此外，您必须有一种方法来自动运行命令。 Glue Crawler 可以配置触发器。