gpt4 book ai didi

Continuous copy or replication from MongoDB Atlas to Google BigQuery(从MongoDB Atlas到Google BigQuery的连续复制)

转载 作者:bug小助手 更新时间:2023-10-22 18:18:30 25 4
gpt4 key购买 nike



We have IoT data coming into MongoDB Atlas and will need to continuously copy/replicate it into Google BigQuery. Trying to find a way that does not involve 3rd party SAAS solution due to budget constraints. We'll need to copy the whole inserted documents and store them in BigQuery as the raw data. If anyone has done this before, kindly share what you have used or done.

我们有物联网数据进入MongoDB Atlas,需要不断地将其复制到Google BigQuery中。由于预算限制,试图找到一种不涉及第三方SAAS解决方案的方法。我们需要复制整个插入的文档,并将它们作为原始数据存储在BigQuery中。如果有人以前这样做过,请分享你使用过或做过的事情。


I have looked into this article "A data pipeline for MongoDB Atlas and BigQuery using Dataflow" from Google and have implemented: the section "#1.MongoDB to BigQuery template" but it copies all existing data then completes the job; as well as the section "#3. MongoDB to BigQuery CDC template" but this doesn't seem to capture any of the incoming data.

我看过谷歌的这篇文章“使用数据流的MongoDB Atlas和BigQuery的数据管道”,并实现了:“#1.MongoDB到BigQuery模板”部分,但它复制了所有现有数据,然后完成了任务;以及“#3。MongoDB到BigQuery CDC模板”,但这似乎没有捕获任何传入数据。


Any advice is greatly appreciated. Thanks.

如有任何建议,不胜感激。谢谢


更多回答

Hi @LearnItDom, Have you tried using the `Mongoexpert` , as mentioned in the Method : 2 of this document ? This might help you.

嗨@LearnIdom,你试过使用本文档方法2中提到的“Mongoexpert”吗?这可能会对你有所帮助。

Hi @LearnItDom, If my answer addressed your question, do consider accepting and upvoting it as per Stack Overflow guidelines, helping more Stack contributors with their research. If not, let me know so that I can improve my answer.

嗨@LearnIdom,如果我的回答解决了你的问题,请考虑根据Stack Overflow指南接受并投票,帮助更多Stack贡献者进行研究。如果没有,请告诉我,这样我就可以改进我的答案。

Thanks for the info, Diraj. I am looking for a more real time process so I am exploring using code now to watch mongodb’s change stream. Will update what I end up using once done.

谢谢你的消息,迪拉杰。我正在寻找一个更实时的过程,所以我现在正在探索使用代码来观察mongodb的更改流。完成后将更新我最终使用的内容。

优秀答案推荐

You can try using the Mongoexpert as mentioned in Method : 2 of this document.

您可以尝试使用本文方法2中提到的Mongoexpert。


Using the Mongoexpert, you can produce a JSON or CSV export of data stored in a MongoDB instance. You can follow the detailed steps in this documentation.

使用Mongoexpert,您可以生成存储在MongoDB实例中的数据的JSON或CSV导出。您可以按照本文档中的详细步骤进行操作。


更多回答

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com