linux - aws s3api list-objects 返回了多少个对象？

Question

我正在使用:

aws s3api list-objects --endpoint-url https://my.end.point/ --bucket my.bucket.name --query 'Contents[].Key' --output text

获取存储桶中的文件列表。

aws s3api 列表对象 documentation page说这个命令最多只返回 1000 个对象，但是我注意到在我的例子中它返回了我存储桶中所有文件的名称。例如，当我运行以下命令时:

aws s3api list-objects --endpoint-url https://my.end.point/ --bucket my.bucket.name --query 'Contents[].Key' --output text | tr "\t" "\n" | wc -l

我得到 13512 显示，这意味着返回了超过 13000 个文件名。

我错过了什么吗？

我使用以下 aws cli 版本:

aws-cli/1.10.57 Python/2.7.3 Linux/3.2.0-4-amd64 botocore/1.4.47

Answer 1

Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. [1]

我认为文档描述中的“(最多 1000)”部分极具误导性。它指的是 cli 发送的每个基础 HTTP 请求的最大页面大小。 --page-size 选项的文档清楚地说明了这一点:

The size of each page to get in the AWS service call. This does not affect the number of items returned in the command's output. Setting a smaller page size results in more calls to the AWS service, retrieving fewer items in each call. This can help prevent the AWS service calls from timing out.

阅读有关分页的 AWS 文档 [2] 时，它变得更加清晰，其中描述了:

For commands that can return a large list of items, the AWS Command Line Interface (AWS CLI) adds three options that you can use to control the number of items included in the output when the AWS CLI calls a service's API to populate the list.

By default, the AWS CLI uses a page size of 1000 and retrieves all available items. For example, if you run aws s3api list-objects on an Amazon S3 bucket that contains 3,500 objects, the CLI makes four calls to Amazon S3, handling the service-specific pagination logic for you in the background and returning all 3,500 objects in the final output.

正如 Ankit 已经正确指出的那样，使用 --max-items 选项是限制结果和停止自动分页的正确解决方案:

To include fewer items at a time in the AWS CLI output, use the --max-items option. The AWS CLI still handles pagination with the service as described above, but prints out only the number of items at a time that you specify. [2]

引用资料

[1] https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects.html
[2] https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html