gpt4 book ai didi

node.js - 如何优化 MongoDB 变更流?

转载 作者:行者123 更新时间:2023-12-05 03:20:29 24 4
gpt4 key购买 nike

我有一些返回 1000 个结果的 mongodb 查询,当我查看 mongodb.com 分析器时,它向我显示:

{
"command": {
"getMore": 8223354687588024000,
"collection": "reservations",
"batchSize": 1000,
"lsid": {
"id": {
"$binary": {
"base64": "n8eH91eURw+xpT6fNEPURQ==",
"subType": "04"
}
}
},
"$clusterTime": {
"clusterTime": {
"$timestamp": {
"t": 1659066401,
"i": 542
}
},
"signature": {
"hash": {
"$binary": {
"base64": "PHh4eHh4eD4=",
"subType": "00"
}
},
"keyId": 7090947493382324000
}
},
"$db": "superhosttools"
},
"originatingCommand": {
"aggregate": "reservations",
"pipeline": [
{
"$changeStream": {
"fullDocument": "updateLookup"
}
}
],
"cursor": {},
"lsid": {
"id": {
"$binary": {
"base64": "n8eH91eURw+xpT6fNEPURQ==",
"subType": "04"
}
}
},
"$clusterTime": {
"clusterTime": {
"$timestamp": {
"t": 1659064839,
"i": 29
}
},
"signature": {
"hash": {
"$binary": {
"base64": "PHh4eHh4eD4=",
"subType": "00"
}
},
"keyId": 7090947493382324000
}
},
"$db": "superhosttools"
},
"planSummary": [
{
"COLLSCAN": {}
}
],
"cursorid": 8223354687588024000,
"keysExamined": 0,
"docsExamined": 26879,
"numYields": 210,
"nreturned": 1000,
"reslen": 15283228,
"locks": {
"ReplicationStateTransition": {
"acquireCount": {
"w": 2206
}
},
"Global": {
"acquireCount": {
"r": 2206
}
},
"Database": {
"acquireCount": {
"r": 2206
}
},
"Collection": {
"acquireCount": {
"r": 1994
}
},
"Mutex": {
"acquireCount": {
"r": 1996
}
},
"oplog": {
"acquireCount": {
"r": 211
}
}
},
"storage": {
"data": {
"bytesRead": 2083760,
"timeReadingMicros": 4772
}
},
"protocol": "op_msg",
"millis": 249,
"v": "4.2.21"
}

这看起来是有趣的部分,我们使用更改流,但我不知道为什么我们会得到 1000 个结果:

    "pipeline": [
{
"$changeStream": {
"fullDocument": "updateLookup"
}
}
],

我正在尝试优化我的 mongodb 服务器。感谢您提供有关如何使此查询更高效的任何帮助。

更新#1

我从 watch 代码中删除了 {"fullDocument": "updateLookup"} 参数,这似乎有所帮助,但我仍然收到一些返回 1000 个文档的类似查询:

    "aggregate": "reservations",
"pipeline": [
{
"$changeStream": {
"fullDocument": "default"
}
}
],

我现在使用以下代码来实现更改流:

Reservation.watch([]).on("change", async (change: ChangeEvent<ReservationDocument>) => {...});

我不知道是否应该向 .watch([]) 调用添加查询以限制文档数量?什么是变更流的最佳实践?

最佳答案

I don’t know why we would get 1000 results

batchSize 将您的游标限制为 1000 个结果.您可以将 batchSize 作为可选参数添加到 collection.watch()调用:

db.collection.watch([], {batchSize: <number>})

由于更改流是游标,您还可以应用 limit()在检索文档之前到光标:

db.collection.watch(pipeline, options).limit(<number>)

Should I add a query to the .watch([]) call to limit the number of documents? What is considered best practices with change streams?

您可能希望将其过滤为您关心的事件(见下文)。除此之外,默认值是最佳实践。

I'm trying to optimize my mongodb server. Any help how to make this query more efficient is appreciated.

optimizing a MongoDB server的问题in general 对于这个问题来说太过板了,所以我会把这个响应限制在更改流上,因为这似乎是 OP 询问的特定用例。

从 MongoDB 5.1 开始,更改流得到了优化,提供了更高效的资源利用并加快了某些聚合管道阶段的执行速度。如果您尚未使用较新的版本,则更新到 5.1 或更高版本将提供性能提升。

你可以看看Change Streams Production Recommendations查看您是否遵守 Mongo 官方建议。与性能唯一相关的部分是这个:

If a sharded collection has high levels of activity, the mongos may not be able to keep up with the changes across all of the shards. Consider utilizing notification filters for these types of collections. For example, passing a $match pipeline configured to filter only insert operations.

因此,如果您在非常活跃的集合上使用 collection.watch() 并且您只需要对某些更改采取行动,请使用 $match筛选出您关心的更改。

例如,如果您只关心“dave”创作的项目:

db.collection.watch(
[ { $match : { author : "dave" } } ]
);

关于批量大小的更多考虑

至于您之前关于批处理大小的问题,减少批处理大小不会对性能产生真正的影响(下面提到的一个异常(exception)),因此如果性能是您唯一关心的问题,您会想看看其他地方。来自docs :

Specifies the number of documents to return in each batch of the response from the MongoDB instance. In most cases, modifying the batch size will not affect the user or the application, as mongosh and most drivers return results as if MongoDB returned a single batch.

有一个警告,解释得很详细here :

If you’re using MongoDB Change Streams and filtering for events that occur infrequently (compared to other activity within the oplog) resuming the change stream may appear “sluggish” using the defaults. Consider specifying a custom batchSize based on your workload to potentially improve the time to returning the first event.

关于node.js - 如何优化 MongoDB 变更流?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/73161572/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com