javascript - 在 Node JS 中缓冲流中的数据以执行批量插入

Question

如何在 nodeJS 中有效地缓冲从流到批量插入的事件，而不是从流中接收到的每条记录的唯一插入。这是我想到的伪代码:

// Open MongoDB connection

mystream.on('data', (record) => {
   // bufferize data into an array
   // if the buffer is full (1000 records)
   // bulk insert into MongoDB and empty buffer
})

mystream.on('end', () => {
   // close connection
})

这看起来很现实吗？
有没有可能的优化？现有的图书馆有利于这一点吗？

Answer 1

使用 NodeJS 的 stream库，这可以简洁有效地实现为:

const stream = require('stream');
const util = require('util');
const mongo = require('mongo');

const streamSource; // A stream of objects from somewhere

// Establish DB connection
const client = new mongo.MongoClient("uri");
await client.connect();

// The specific collection to store our documents
const collection = client.db("my_db").collection("my_collection");

await util.promisify(stream.pipeline)( 
  streamSource, 
  stream.Writable({
    objectMode: true,
    highWaterMark: 1000,
    writev: async (chunks, next) => {
      try {
        const documents = chunks.map(({chunk}) => chunk);
        
        await collection.insertMany(docs, {ordered: false});

        next();
      }
      catch( error ){
        next( error );
      }
    }
  })
);