java - 有没有办法在用 Java 创建文件时将文件发送到 AWS S3 存储桶？

Question

我有来自数据库的传入数据，应将其写入 CSV 文件，该文件应再次存储在 Amazon S3 存储桶中。我不允许使用太多本地存储空间(大约 1GB)。是否可以在不使用本地存储或仅使用我拥有的少量存储的情况下将传入数据作为 CSV 文件上传？该文件将超过 10 GB。

Answer 1

使用AWS SDK非常容易做到，但关键是在开始上传之前您需要知道文件大小

如果您知道文件有多大，那么您可以准备自己的 InputStream 并将其传递给 S3 客户端，如下所示:

public class DynamicUpload {

    public static void main(String[] args) {
        // Create S3 client
        AmazonS3 s3 = AmazonS3Client.builder().withRegion("eu-central-1").build();
        CsvStream stream = new CsvStream();
        // When providing InputStream, you must set content length
        ObjectMetadata obj = new ObjectMetadata();
        obj.setContentLength(stream.getSize());
        obj.setContentType("text/plain");
        // Pass created InputStream as a source
        s3.putObject(new PutObjectRequest("files.stirante.com", "stackOverflow.csv", stream, obj));
    }

    private static class CsvStream extends InputStream {

        private static DecimalFormat format = new DecimalFormat("00");
        // Target size for testing purposes
        private int size = 100000;
        // This is size of one row "XX;XX;XX\n"
        private int itemSize = 9;
        // Since we increment it at the very beginning, we set it to -1
        private int currentItemIndex = -1;
        // Current row, we're returning
        private byte[] currentItem = null;
        // Byte index in current row
        private int currentItemByteIndex = 0;

        /**
         * Returns size of the whole file
         */
        public int getSize() {
            return size * itemSize;
        }

        @Override
        public int read() throws IOException {
            // Every time read is called, we return another character from created earlier row
            currentItemByteIndex++;
            // If row is not initialized or earlier row was already fully returned, we create another row
            if (currentItem == null || currentItemByteIndex >= itemSize) {
                currentItemIndex++;
                // If we don't have another row, we throw end of file exception
                if (currentItemIndex == size) {
                    throw new EOFException();
                }
                // Format guarantees us, that in case of number smaller than 10, it will still return 2 characters (e.g. 02)
                String s = format.format(Math.random() * 99) + ";" +
                        format.format(Math.random() * 99) + ";" +
                        format.format(Math.random() * 99) + "\n";
                currentItem = s.getBytes();
                currentItemByteIndex = 0;
            }
            return currentItem[currentItemByteIndex];
        }
    }
}

Example generated file

文档:PutObjectRequest