javascript - 在 javascript 中写入数百万行 csv 的最佳方法

Question

我有一个巨大的数组，其中包含大约 1000 万个对象，每个对象都有 11 个不同的键值对。

将数组内容写入 csv 的最佳方法是什么？

我尝试使用 csv-writer 和 fast-csv - Reference

使用 csv 编写器

fastcsv  
  .write(final, { headers: true })
  .pipe(ws);

使用快速 csv

csvWriter.writeRecords(final)
    .then(() => {
      console.log('...Writing csv Done. Check CSV');
    });

这两种方法都会导致无效的字符串长度

.join(RECORD_DELIMITER) + RECORD_DELIMITER; ^

RangeError: Invalid string length at Array.join (native)

示例对象如下所示

{ distance: 0.14,
a_id: 1923,
long: -122.234,
lat: 47.631,
DPlong: -122.234,
DPlat: 47.632,
class: 'secondary',
way_id: 2,
timestamp: '5-6-2017',
user: 'hello',
code: 'DS' }

解决它的最佳方法是什么？任何帮助将不胜感激。谢谢。

Answer 1

您不能将整个数组转换为字符串，因为字符串太长而无法处理 js，而是必须按 block 格式化:

const tick = () => new Promise(res => setTimeout(res, 0));

const toRow = obj =>
  obj.distance + "," +
  obj.a_id + "," + 
  obj.long + "," +
  obj.lat + "," +
  obj.DPlong + "," +
  obj.DPlat + "," +
  obj.class + "," +
  obj.way_id + "," + 
  obj.timestamp + "," + 
  obj.user + "," + 
  obj.code;

const formatChunk = array => array.map(toRow).join("\n") + "\n";

const size = 1000; // <- experiment with it
(async function() {
  for(let i = 0; i < final.length; i += size) {
    ws.write(formatChunk(final.slice(i, i + size)));
    await tick();
  }
  ws.end();
})();