r - 在 R( future )中进行多处理时如何避免填充 RAM？

Question

我正在使用 furrr，它建立在 future 之上。

我有一个非常简单的问题。我有一个文件列表，比如说 list('/mydata/file1.csv.gz', '/mydata/file1.csv.gz') 我正在用一个简单的函数并行处理它们加载数据，进行一些过滤，然后将其写入磁盘。

本质上，我的功能是

processing_func <- function(file){
  mydata <- readr::read_csv(file)
  mydata <- mydata %>% dplyr::filter(var == 1)
  data.table::fwrite(mydata, 'myfolder/processed.csv.gz')
  rm()
  gc()
}

所以我只是在运行

listfiles %>% furrr::future_map(., processing_func(.x))

这是有效的，但是尽管我调用了 gc() 和 rm()，RAM 一直在填满，直到 session 崩溃。

这里的概念问题是什么？当我明确丢弃它们时，为什么一些残留对象会以某种方式保留在内存中？

谢谢!

Answer 1

您可以尝试使用 callr future 的计划，它可能会减少内存饥饿。正如引用自 future.callr vignette

When using callr futures, each future is resolved in a fresh background R session which ends as soon as the value of the future has been collected. In contrast, multisession futures are resolved in background R worker sessions that serve multiple futures over their life spans. The advantage with using a new R process for each future is that it is that the R environment is guaranteed not to be contaminated by previous futures, e.g. memory allocations, finalizers, modified options, and loaded and attached packages. The disadvantage, is an added overhead of launching a new R process

library("future.callr")
plan(callr)