javascript - NodeJs 在外部数据请求时内存不足

Question

我正在使用 Axios 在后端执行大型数据提取:

await Axios({
      method: 'GET',
      url,
      headers: {
        'content-type': 'application/json',
        Authorization: `Bearer ${forgeAccessToken}`,
      },
    });

NodeJS 崩溃。内存问题:

  1|rest-backenq  | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

我将内存从 4GB 增加到 16GB。

没有效果。

我无法控制从中获取数据的外部 API。

我该怎么做才能解决这个问题？

Answer 1

我认为你的文件正在下载到你的内存中，使用 request 模块而不是 Axios 来下载文件

var fs = require('fs');
var request = require('request');
var progress = require('request-progress');

// The options argument is optional so you can omit it 
progress(request({
    url : url,
    headers: {
        'content-type': 'application/json',
        'Authorization': `Bearer ${forgeAccessToken}`
    }
}), {
    // throttle: 2000,                    // Throttle the progress event to 2000ms, defaults to 1000ms 
    // delay: 1000,                       // Only start to emit after 1000ms delay, defaults to 0ms 
    // lengthHeader: 'x-transfer-length'  // Length header to use, defaults to content-length 
})
.on('progress', function (state) {
    // The state is an object that looks like this: 
    // { 
    //     percent: 0.5,               // Overall percent (between 0 to 1) 
    //     speed: 554732,              // The download speed in bytes/sec 
    //     size: { 
    //         total: 90044871,        // The total payload size in bytes 
    //         transferred: 27610959   // The transferred payload size in bytes 
    //     }, 
    //     time: { 
    //         elapsed: 36.235,        // The total elapsed seconds since the start (3 decimals) 
    //         remaining: 81.403       // The remaining seconds to finish (3 decimals) 
    //     } 
    // } 
    console.log('progress', state);
})
.on('error', function (err) {
    // Do something with err 
})
.on('end', function () {
    // Do something after request finishes 
})
.pipe(fs.createWriteStream('IE11.Win8.1.For.Windows.VirtualBox.zip'));

有关更多信息，请关注此问题， What is way to download big file in NodeJS?