javascript - fatal error : Scavenger: semi-space copy Allocation failed

Question

我在运行 Node 应用程序时遇到错误，该 Node 应用程序处理正在读入数组的非常大的文件(7600 万条记录)。

Suse 服务器有 64GB 的内存，我也有 100GB 的交换文件。然而，堆转储发生在大约 30GB 的内存中。读取了大约 3100 万条记录。

Node 版本为 12.22.1

这是一个分析应用程序，它是初始提取数据的一次性工作负载。应用程序一次以 100k 条记录为单位读取文件。

我已经尝试了这些参数的多种排列，这是最后一次运行。

--max-old-space-size=196608
--max-semi-space-size=256
--semi-space-growth-factor=4

我以前在 Node 4 上仅使用 --max-old-space-size 运行这种大型数据集没有问题，但从我读到的内容来看，内存管理与 Node 12 以后不同。

感谢您提供的任何指导或建议。

感谢和问候，

迪兰克

^[[0;34mINFO ^[[0m      2021-04-27 08:10:50     ^[[0;1mDSParser^[[0m    Read 31700000 lines
                ^[[0;4mtable^[[0m: "ekpo"
                ^[[0;4mstep^[[0m: 0
                ^[[0;4mmemory^[[0m: {"rss":21905813504,"heapTotal":17302638592,"heapUsed":17253034680,"external":4473602959,"arrayBuffers":4472358538}
^[[0;34mINFO ^[[0m      2021-04-27 08:10:56     ^[[0;1mDSParser^[[0m    Read 31800000 lines
                ^[[0;4mtable^[[0m: "ekpo"
                ^[[0;4mstep^[[0m: 0
                ^[[0;4mmemory^[[0m: {"rss":22110760960,"heapTotal":17492168704,"heapUsed":17435101184,"external":4473635727,"arrayBuffers":4472391306}

<--- Last few GCs --->

[11070:0x4c9f3c0]  1092787 ms: Scavenge 16733.7 (16784.8) -> 16733.6 (16784.8) MB, 84.5 / 0.0 ms  (average mu = 0.988, current mu = 0.990) allocation failure
[11070:0x4c9f3c0]  1092869 ms: Scavenge 16733.7 (16784.8) -> 16733.6 (16784.8) MB, 80.9 / 0.0 ms  (average mu = 0.988, current mu = 0.990) allocation failure
[11070:0x4c9f3c0]  1092953 ms: Scavenge 16733.7 (16784.8) -> 16733.7 (16784.8) MB, 83.9 / 0.0 ms  (average mu = 0.988, current mu = 0.990) allocation failure


<--- JS stacktrace --->
Cannot get stack trace in GC.

>  FATAL ERROR: Scavenger: semi-space copy Allocation failed -
> JavaScript heap out of memory  1: 0xa1a640 node::Abort()
> [/home/forestpin/data/retail/bin/node/node/bin/node]  2: 0xa1aa4c
> node::OnFatalError(char const*, char const*)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  3: 0xb9a68e
> v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  4: 0xb9aa09
> v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char
> const*, bool) [/home/forestpin/data/retail/bin/node/node/bin/node]  5:
> 0xd57c85  [/home/forestpin/data/retail/bin/node/node/bin/node]  6:
> 0xdba162  [/home/forestpin/data/retail/bin/node/node/bin/node]  7:
> 0xdbbfb6 void
> v8::internal::BodyDescriptorBase::IteratePointers<v8::internal::IterateAndScavengePromotedObjectsVisitor>(v8::internal::HeapObject,
> int, int, v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  8: 0xdbc680 void
> v8::internal::BodyDescriptorBase::IterateJSObjectBodyImpl<v8::internal::IterateAndScavengePromotedObjectsVisitor>(v8::internal::Map,
> v8::internal::HeapObject, int, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node]  9: 0xdbcfd1 void
> v8::internal::BodyDescriptorApply<v8::internal::CallIterateBody, void,
> v8::internal::Map, v8::internal::HeapObject, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*>(v8::internal::InstanceType,
> v8::internal::Map, v8::internal::HeapObject, int,
> v8::internal::IterateAndScavengePromotedObjectsVisitor*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 10: 0xdbd87e
> v8::internal::Scavenger::IterateAndScavengePromotedObject(v8::internal::HeapObject,
> v8::internal::Map, int)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 11: 0xdbdb8b
> v8::internal::Scavenger::Process(v8::internal::OneshotBarrier*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 12: 0xdc15bd
> v8::internal::ScavengingTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 13: 0xd75885
> v8::internal::ItemParallelJob::Task::RunInternal()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 14: 0xd75c94
> v8::internal::ItemParallelJob::Run()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 15: 0xdbf65a
> v8::internal::ScavengerCollector::CollectGarbage()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 16: 0xd50741
> v8::internal::Heap::Scavenge()
> [/home/forestpin/data/retail/bin/node/node/bin/node] 17: 0xd64f1b
> v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
> v8::GCCallbackFlags)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 18: 0xd65a85
> v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
> v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 19: 0xd6853c
> v8::internal::Heap::AllocateRawWithRetryOrFail(int,
> v8::internal::AllocationType, v8::internal::AllocationOrigin,
> v8::internal::AllocationAlignment)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 20: 0xd2ef5b
> v8::internal::Factory::NewFillerObject(int, bool,
> v8::internal::AllocationType, v8::internal::AllocationOrigin)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 21: 0x107158e
> v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*,
> v8::internal::Isolate*)
> [/home/forestpin/data/retail/bin/node/node/bin/node] 22: 0x140de99 
> [/home/forestpin/data/retail/bin/node/node/bin/node]

Answer 1

我也遇到了同样的错误，下面的操作系统参数对我有用。

sudo sysctl -w vm.max_map_count=655300 //10times bigger than default

下面是我的错误信息。

<--- Last few GCs --->  
[356958:0x57dbc60]   214270 ms: Scavenge 17575.0 (17613.0) -> 17574.8 (17613.0) MB, 1010.9 / 0.0 ms  (average mu = 0.930, current mu = 0.931) task                              
[356958:0x57dbc60]   215347 ms: Scavenge 17575.7 (17613.0) -> 17575.5 (17613.0) MB, 1073.1 / 0.0 ms  (average mu = 0.930, current mu = 0.931) task                            
[356958:0x57dbc60]   216475 ms: Scavenge 17576.4 (17613.0) -> 17576.3 (17613.0) MB, 1124.9 / 0.0 ms  (average mu = 0.930, current mu = 0.931) task                              
<--- JS stacktrace --->                                                                                                                                                       
FATAL ERROR: Scavenger: semi-space copy Allocation failed - JavaScript heap out of memory   
scripts/MY_SCRIPT.sh: line 5: 356958 Segmentation fault      (core dumped) node --max-old-space-size=50000 'MY_NODE_SCRIPT.js'