r - R (windows) 中大栅格的并行处理

Question

我正在使用 doSNOW 包，更具体地说，是 parLapply 函数对大型栅格数据集列表(操作系统:Windows x64)执行重新分类(以及随后的其他操作)。

代码看起来有点像这个简约的示例:

library(raster)
library(doSNOW)

#create list containing test rasters

x <- raster(ncol=10980,nrow=10980) 
x <- setValues(x,1:ncell(x)) 

list.x <- replicate( 9 , x )

#setting up cluster

NumberOfCluster <- 8
cl <- makeCluster(NumberOfCluster)
registerDoSNOW(cl)
junk <- clusterEvalQ(cl,library(raster))

#perform calculations on each raster

list.x <- parLapply(cl,list.x,function(x) calc(x,function(x) { x * 10 }))

#stop cluster

stopCluster(cl)

该代码实际上按预期工作。当我想继续处理结果时，就会出现问题。我收到此错误消息:

> plot(list.x[[1]])
Error in file(fn, "rb") : cannot open the connection
In addition: Warning message:
In file(fn, "rb") :
  cannot open file 'C:\Users\*****\AppData\Local\Temp\RtmpyKYdpY\raster\r_tmp_2016-02-29_133158_752_67867.gri': No such file or directory

据我了解，由于栅格很大，因此它们保存在磁盘上的临时文件中。当我关闭 Snow Cluster 时，这些文件将无法再访问。

所以我的问题是，集群关闭后如何访问数据？我可以继续使用此方法吗？

谢谢!

Answer 1

您可以将特定的文件名传递给calc(或者，例如reclassify)，并让您的函数将这些文件名作为向量返回以读入堆栈:

ff <- parSapply(cl, list.x, function(x) { 
  calc(x, function(x) x*10, filename=f <- tempfile(fileext='.tif'))
  f
})

s <- stack(ff)

但还要看看 ?clusterR - 我怀疑它可以与 reclassify 一起使用。来自文档:

This function only works with functions that have a Raster* object as first argument and that operate on a cell by cell basis (i.e., there is no effect of neigboring cells) and return an object with the same number of cells as the input raster object. The first argument of the function called must be a Raster* object. There can only be one Raster* object argument. For example, it works with calc and it also works with overlay as long as you provide a single RasterStack or RasterBrick as the first argument.