javascript - 列出 Google Cloud 存储桶中包含数千个文件的文件。

Question

我正在运行 Node.js 脚本来获取 Google Cloud Storage 上存储桶中的文件数量。

在一个包含大约 30K 文件的存储桶中，我在几秒钟内就得到了结果。在包含大约 300K 文件的存储桶中，我收到以下错误:

<--- Last few GCs --->

[10508:0000014DB738ADB0]  2053931 ms: Mark-sweep 1400.6 (1467.7) -> 1400.6 (1437.2) MB, 1292.2 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1292 ms) last resort GC in old space requested
[10508:0000014DB738ADB0]  2055233 ms: Mark-sweep 1400.6 (1437.2) -> 1400.6 (1437.2) MB, 1301.9 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001A6B8025EE1 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [D:\Libraries\Documents\project-name\node_modules\@google-cloud\storage\src\acl.js:~717] [pc=0000005E62D95DCF](this=0000016DB7602311 <undefined>,accessMethod=0000016DB7602AC1 <String[3]: add>)
    2: arguments adaptor frame: 3->1
    3: forEach(this=00000335A20E8891 <JSArray[2]>)
    4: /* anonymous */(a...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

下面是我正在使用的代码。有更好的办法吗？

const Storage = require('@google-cloud/storage');

function listFiles(bucketName) {
  // [START storage_list_files]
  // Imports the Google Cloud client library


  // Creates a client
  const storage = new Storage();

  /**
   * TODO(developer): Uncomment the following line before running the sample.
   */
  // const bucketName = 'Name of a bucket, e.g. my-bucket';

  // Lists files in the bucket
  return storage
    .bucket(bucketName)
    .getFiles(); ///const files = results[0];
  // [END storage_list_files]
}

listFiles('bucket-name')
    .then(x => {
        console.log('Number of files: ', x[0].length)
    });

Answer 1

大多数返回列表的方法都提供该方法的流版本。在这种情况下，您需要使用 bucket.getFilesStream()

bucket.getFilesStream()
  .on('error', console.error)
  .on('data', function(file) {
    // file is a File object.
  })
  .on('end', function() {
    // All files retrieved.
  });

或者，您可以通过结果禁用自动分页和手册页

const callback = function(err, files, nextQuery, apiResponse) {
  if (nextQuery) {
    // More results exist.
    bucket.getFiles(nextQuery, callback);
  }
};

bucket.getFiles({
  autoPaginate: false
}, callback);