amazon-web-services - 从技术上讲，s3n、s3a 和 s3 之间有什么区别？

Question

我知道 https://wiki.apache.org/hadoop/AmazonS3 的存在以及以下的话:

S3 Native FileSystem (URI scheme: s3n) A native filesystem for reading and writing regular files on S3. The advantage of this filesystem is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. The disadvantage is the 5GB limit on file size imposed by S3.

S3A (URI scheme: s3a) A successor to the S3 Native, s3n fs, the S3a: system uses Amazon's libraries to interact with S3. This allows S3a to support larger files (no more 5GB limit), higher performance operations and more. The filesystem is intended to be a replacement for/successor to S3 Native: all objects accessible from s3n:// URLs should also be accessible from s3a simply by replacing the URL schema.

S3 Block FileSystem (URI scheme: s3) A block-based filesystem backed by S3. Files are stored as blocks, just like they are in HDFS. This permits efficient implementation of renames. This filesystem requires you to dedicate a bucket for the filesystem - you should not use an existing bucket containing files, or write other files to the same bucket. The files stored by this filesystem can be larger than 5GB, but they are not interoperable with other S3 tools.

为什么 URI 上的字母变化会造成如此大的不同？例如

val data = sc.textFile("s3n://bucket-name/key")

到

val data = sc.textFile("s3a://bucket-name/key")

此更改背后的技术差异是什么？有没有我可以阅读的好文章？

Answer 1

URI 方案上的字母变化会产生很大的不同，因为它会导致使用不同的软件连接到 S3。有点像 http 和 https 之间的区别 - 它只是一个字母的变化，但它会引发很大的行为差异。

s3 和 s3n/s3a 之间的区别在于 s3 是在 Amazon S3 之上基于 block 的覆盖，而 s3n/s3a 不是(它们是基于对象的)。

s3n和s3a的区别在于，s3n支持最大5GB的对象，而s3a支持最大5TB的对象，性能更高(都是因为它使用了分片上传)。 s3a 是 s3n 的继承者。

根据 Work with Storage and File Systems ，使用 EMRFS 时:

Previously, Amazon EMR used the s3n and s3a file systems. While both still work, we recommend that you use the s3 URI scheme for the best performance, security, and reliability.

关于 s3n 和 s3a 的其他历史引用可以在 this article 找到来自亚马逊(仅适用于 wayback 机器)。