javascript - 将数据导入 mongodb 时如何修复 JavaScript 堆内存不足错误？

Question

任何人都可以为我指出正确的方向，为什么我无法将数据导入到 mongodb？当我尝试仅导入整个文件的前 100 行时，我得到

database-operations git:(master) ✗ node import_acparts_to_mongdb.js (node:10216) Warning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increas e limit ➜ database-operations git:(master) ✗

我尝试从同一个文件(一个具有以下结构的 csv 文件)导入 600.000 行:

设施；元素编号；零件名称；零件描述；净重；海关统计PBL;5535210444;封面;封面;0;84314980D37;5535211545;支架;支架-消防补给 jar A101-20;2,939;72169110PBL;5535211234;支架;支架-消防补给 jar A101-20;2,939;84314300PBL;5535212478;环扣;环扣;0,045;84314980…………......

➜ database-operations git:(master) ✗ node import_acparts_to_mongdb.js

<--- Last few GCs --->

38787 ms: Mark-sweep 1384.9 (1436.8) -> 1384.8 (1436.8) MB, 1181.9 / 0.0 ms [allocation failure] [GC in old space requested]. 39964 ms: Mark-sweep 1384.8 (1436.8) -> 1384.8 (1436.8) MB, 1177.7 / 0.0 ms [allocation failure] [GC in old space requested]. 41199 ms: Mark-sweep 1384.8 (1436.8) -> 1385.8 (1420.8) MB, 1234.0 / 0.0 ms [last resort gc]. 42429 ms: Mark-sweep 1385.8 (1420.8) -> 1386.9 (1420.8) MB, 1229.8 / 0.0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x4962c9cfb39 1: $__validate [/Users/isaklafleur/Dropbox/Isak/Coding/Other/autoMDM/node_modules/mongoose/lib/document.js:~1404] [pc=0xe52ebc4f d97] (this=0x383867c1f221 ,callback=0x383867c201e1 ) 2: validate [/Users/isaklafleur/Dropbox/Isak/Coding/Other/autoMDM/node_modules/mongoose/lib/document.js:~1324] [pc=0x...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 1: node::Abort() [/usr/local/bin/node] 2: node::FatalException(v8::Isolate*, v8::Local, v8::Local) [/usr/local/bin/node] 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/local/bin/node] 4: v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/local/bin/node] 5: v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/local/bin/node] 6: 0xe52eb8079a7 [1] 10085 abort node import_acparts_to_mongdb.js ➜ database-operations git:(master) ✗

<小时/>

const mongoose  = require('mongoose'),
    parse       = require('csv-parse'),
    path        = require('path'),
    fs          = require('fs'),
    ACpart      = require('./models/acparts');

mongoose.Promise = require('bluebird');

mongoose.connect('mongodb://localhost/automdm_test');

const db = mongoose.connection;

db.on('error', console.error.bind(console, 'connection error:'));

db.once('open', function() {
    // we're connected!

    const p = path.join(__dirname, '/../', 'file-operations', 'csv-files');
    //console.log(p);

    const parser = parse({delimiter: ';'}, function(err, data){
        //console.log(data);
        const facility = data.map((item,i) => data[i][0]);
        const item_number = data.map((item,i) => data[i][1]);
        const part_name = data.map((item,i) => data[i][2]);
        const part_description = data.map((item,i) => data[i][3]);
        const net_weight = data.map((item,i) => data[i][4]);
        const customs_statistical = data.map((item,i) => data[i][5]);

        // Looping and storing the data into mongodb
        for (let i = 1; i < data.length; i++) {

            const newACpart = new ACpart();
            newACpart.facility = facility[i]
            newACpart.item_number = item_number[i];
            newACpart.part_name = part_name[i];
            newACpart.part_description = part_description[i];
            newACpart.net_weight = net_weight[i];
            newACpart.customs_statistical = customs_statistical[i];
            newACpart.save()
            .then(function() {
                mongoose.disconnect();
            })
            .catch(function(err) {
                console.log('There was an error', err);
            });
        }
    });
    fs.createReadStream(p + '/mrsparts.csv').pipe(parser);
});

Answer 1

如果内存大于堆，您将无法将所有内容放入内存中。使用流式 CSV 解析器，例如其中之一:

• https://www.npmjs.com/package/csv-stream
• https://www.npmjs.com/package/node-stream-csv
• https://www.npmjs.com/package/stream-csv-enhanced
• https://www.npmjs.com/package/csv-parse
• https://www.npmjs.com/package/fast-csv

然后分批发送到数据库，而不是一次全部发送到数据库。