java - AmazonS3.getObject(request).getObjectContent() 频繁出现 NoHttpResponseException

Question

我有一个帮助程序尝试从 S3 进行线程下载。很多时候(大约 1% 的请求)我收到一条关于 NoHttpResponseException 的日志消息，一段时间后从 S3ObjectInputStream 读取时导致 SocketTimeoutException >.

是我做错了什么，还是只是我的路由器/互联网？或者这是 S3 的预期？我没有注意到其他地方的问题。

  public void
fastRead(final String key, Path path) throws StorageException 
    {
        final int pieceSize = 1<<20;
        final int threadCount = 8;

        try (FileChannel channel = (FileChannel) Files.newByteChannel( path, WRITE, CREATE, TRUNCATE_EXISTING ))
        {
            final long size = s3.getObjectMetadata(bucket, key).getContentLength();
            final long pieceCount = (size - 1) / pieceSize + 1;

            ThreadPool pool = new ThreadPool (threadCount);
            final AtomicInteger progress = new AtomicInteger();

            for(int i = 0; i < size; i += pieceSize)
            {
                final int start = i;
                final long end = Math.min(i + pieceSize, size);

                pool.submit(() ->
                {
                    boolean retry;
                    do
                    {
                        retry = false;
                        try
                        {
                            GetObjectRequest request = new GetObjectRequest(bucket, key);
                            request.setRange(start, end - 1);
                            S3Object piece = s3.getObject(request);
                            ByteBuffer buffer = ByteBuffer.allocate ((int)(end - start));
                            try(InputStream stream = piece.getObjectContent())
                            {
                                IOUtils.readFully( stream, buffer.array() );
                            }
                            channel.write( buffer, start );
                            double percent = (double) progress.incrementAndGet() / pieceCount * 100.0;
                            System.err.printf("%.1f%%\n", percent);
                        }
                        catch(java.net.SocketTimeoutException | java.net.SocketException e)
                        {
                            System.err.println("Read timed out. Retrying...");
                            retry = true;
                        }
                    }
                    while (retry);

                });
            }

            pool.<IOException>await();
        }
        catch(AmazonClientException | IOException | InterruptedException e)
        {
            throw new StorageException (e);
        }
    }

2014-05-28 08:49:58 INFO com.amazonaws.http.AmazonHttpClient executeHelper Unable to execute HTTP request: The target server failed to respond
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:95)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:66)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:713)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:518)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:385)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:233)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3569)
at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1130)
at com.syncwords.files.S3Storage.lambda$fastRead$0(S3Storage.java:123)
at com.syncwords.files.S3Storage$$Lambda$3/1397088232.run(Unknown Source)
at net.almson.util.ThreadPool.lambda$submit$8(ThreadPool.java:61)
at net.almson.util.ThreadPool$$Lambda$4/1980698753.call(Unknown Source)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)

Answer 1

更新:针对我在 GitHub 上创建的问题，对 AWS SDK 进行了更新。我不确定情况是如何改变的。这个答案的第二部分(批评 getObject)可能(希望如此？)是错误的。

---

S3 被设计成会失败，而且经常失败。

幸运的是，适用于 Java 的 AWS 开发工具包具有用于重试请求的内置工具。不幸的是，它们没有涵盖下载 S3 对象时出现 SocketExceptions 的情况(它们在上传和执行其他操作时确实起作用)。因此，类似于问题中的代码是必要的(见下文)。

当该机制按预期运行时，您仍会在日志中看到消息。您可以选择通过过滤来自 com.amazonaws.http.AmazonHttpClient 的 INFO 日志事件来隐藏它们。 (AWS SDK 使用 Apache Commons Logging。)

根据您的网络连接和亚马逊服务器的健康状况，重试机制可能会失败。 lvlv指出，配置相关参数的方式是通过ClientConfiguration .我建议更改的参数是重试次数，默认情况下为 3。您可能尝试的其他事情是增加或减少连接和套接字超时(默认 50 秒，这不仅足够长，考虑到无论如何您都会经常超时的事实，它可能太长了)和使用 TCP KeepAlive(默认关闭)。

ClientConfiguration cc = new ClientConfiguration()
    .withMaxErrorRetry (10)
    .withConnectionTimeout (10_000)
    .withSocketTimeout (10_000)
    .withTcpKeepAlive (true);
AmazonS3 s3Client = new AmazonS3Client (credentials, cc);

重试机制甚至可以通过设置 RetryPolicy 来覆盖(同样，在 ClientConfiguration 中)。它最有趣的元素是 RetryCondition，默认情况下:

checks for various conditions in the following order:

• Retry on AmazonClientException exceptions caused by IOException;
• Retry on AmazonServiceException exceptions that are either 500 internal server errors, 503 service unavailable errors, service throttling errors or clock skew errors.

参见 SDKDefaultRetryCondition javadoc和 source

隐藏在 SDK 其他地方的半成功重试工具

内置机制(用于整个 AWS SDK)不处理的是读取 S3 对象数据。

如果您调用 AmazonS3.getObject (GetObjectRequest getObjectRequest, File destinationFile)，AmazonS3Client 会使用自己的重试机制。该机制位于 ServiceUtils.retryableDownloadS3ObjectToFile ( source ) 内部，它使用次优的硬连线重试行为(它只会重试一次，并且永远不会在 SocketException 上重试!)。 ServiceUtils 中的所有代码似乎都设计得很糟糕 (issue)。

我使用的代码类似于:

  public void
read(String key, Path path) throws StorageException
    {
        GetObjectRequest request = new GetObjectRequest (bucket, key);

        for (int retries = 5; retries > 0; retries--) 
        try (S3Object s3Object = s3.getObject (request))
        {
            if (s3Object == null)
                return; // occurs if we set GetObjectRequest constraints that aren't satisfied

            try (OutputStream outputStream = Files.newOutputStream (path, WRITE, CREATE, TRUNCATE_EXISTING))
            {
                byte[] buffer = new byte [16_384];
                int bytesRead;
                while ((bytesRead = s3Object.getObjectContent().read (buffer)) > -1) {
                    outputStream.write (buffer, 0, bytesRead);
                }
            }
            catch (SocketException | SocketTimeoutException e)
            {
                // We retry exceptions that happen during the actual download
                // Errors that happen earlier are retried by AmazonHttpClient
                try { Thread.sleep (1000); } catch (InterruptedException i) { throw new StorageException (i); }
                log.log (Level.INFO, "Retrying...", e);
                continue;
            }
            catch (IOException e)
            {
                // There must have been a filesystem problem
                // We call `abort` to save bandwidth
                s3Object.getObjectContent().abort();
                throw new StorageException (e);
            }

            return; // Success
        }
        catch (AmazonClientException | IOException e)
        {
            // Either we couldn't connect to S3
            // or AmazonHttpClient ran out of retries
            // or s3Object.close() threw an exception
            throw new StorageException (e);
        }

        throw new StorageException ("Ran out of retries.");
    }