amazon-web-services - DynamoDB 表迁移 - 关于保持数据同步的建议？

Question

我的团队正在将多个不同的 DynamoDB 表迁移到一组两个 DynamoDB 表中。本质上，我们正在更改模型并将大量数据合并到两个表中(现在分布在 6 个表中)。

我们正在考虑使用 DynamoDB 流，并让它们触发 Lambda，该 Lambda 处理触发某些写入新表的 API 的逻辑。但是，我也试图弄清楚如何处理旧数据，因为我们还必须将表中已有的所有内容迁移到新表中。

我猜测一个选择是运行一些程序来扫描 DynamoDB 表中的每个项目(有些项目大约有 1 亿个)，并且对于每个项目，调用 Lambda 调用的相同 API 来写入新表到。但是，我不确定当我们在迁移过程中也经常将新记录写入旧表时，对旧表的扫描如何处理。

有人对进行此类迁移并保持同步有一些建议吗？

谢谢!

Answer 1

Streams 和 lambda 是一个不错的 solution (by Abhaya Chauhan) ...
(也是pluralsight上的video)

In this situation, try leveraging DynamoDB Streams and AWS Lambda to remodel data as needed.

A nice way to restructure your table definition is to leverage DynamoDB triggers, following these steps:

1. Create a new table (let us call this NewTable), with the desired key structure, LSIs, GSIs.

2. Enable DynamoDB Streams on the original table

3. Associate a Lambda to the Stream, which pushes the record into NewTable. (This Lambda should trim off the migration flag in Step 5)
4. [Optional] Create a GSI on the original table to speed up scanning items. Ensure this GSI only has attributes: Primary Key, and Migrated (See Step 5).
5. Scan the GSI created in the previous step (or entire table) and use the following Filter: FilterExpression = "attribute_not_exists(Migrated)" Update each item in the table with a migrate flag (ie: “Migrated”: { “S”: “0” }, which sends it to the DynamoDB Streams (using UpdateItem API, to ensure no data loss occurs).

NOTE You may want to increase write capacity units on the table during the updates.

6. The Lambda will pick up all items, trim off the Migrated flag and push it into NewTable.
7. Once all items have been migrated, repoint the code to the new table
8. Remove original table, and Lambda function once happy all is good.