amazon-web-services - 什么是 AWS Request Metrics Full Support？如何关闭它？

Question

我正在尝试将一些数据从 Spark 数据帧保存到 S3 存储桶。这很简单:

dataframe.saveAsParquetFile("s3://kirk/my_file.parquet")

数据已成功保存，但用户界面却忙了很长时间。我得到了数千行:

2015-09-04 20:48:19,591 INFO  [main] amazonaws.latency (AWSRequestMetricsFullSupport.java:log(203)) - StatusCode=[200], ServiceName=[Amazon S3], AWSRequestID=[5C3211750F4FF5AB], ServiceEndpoint=[https://kirk.s3.amazonaws.com], HttpClientPoolLeasedCount=0, RequestCount=1, HttpClientPoolPendingCount=0, HttpClientPoolAvailableCount=1, ClientExecuteTime=[63.827], HttpRequestTime=[62.919], HttpClientReceiveResponseTime=[61.678], RequestSigningTime=[0.05], ResponseProcessingTime=[0.812], HttpClientSendRequestTime=[0.038],
2015-09-04 20:48:19,610 INFO  [main] amazonaws.latency (AWSRequestMetricsFullSupport.java:log(203)) - StatusCode=[204], ServiceName=[Amazon S3], AWSRequestID=[709DA41540539FE0], ServiceEndpoint=[https://kirk.s3.amazonaws.com], HttpClientPoolLeasedCount=0, RequestCount=1, HttpClientPoolPendingCount=0, HttpClientPoolAvailableCount=1, ClientExecuteTime=[18.064], HttpRequestTime=[17.959], HttpClientReceiveResponseTime=[16.703], RequestSigningTime=[0.06], ResponseProcessingTime=[0.003], HttpClientSendRequestTime=[0.046],
2015-09-04 20:48:19,664 INFO  [main] amazonaws.latency (AWSRequestMetricsFullSupport.java:log(203)) - StatusCode=[204], ServiceName=[Amazon S3], AWSRequestID=[1B1EB812E7982C7A], ServiceEndpoint=[https://kirk.s3.amazonaws.com], HttpClientPoolLeasedCount=0, RequestCount=1, HttpClientPoolPendingCount=0, HttpClientPoolAvailableCount=1, ClientExecuteTime=[54.36], HttpRequestTime=[54.26], HttpClientReceiveResponseTime=[53.006], RequestSigningTime=[0.057], ResponseProcessingTime=[0.002], HttpClientSendRequestTime=[0.034],
2015-09-04 20:48:19,675 INFO  [main] amazonaws.latency (AWSRequestMetricsFullSupport.java:log(203)) - StatusCode=[404], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: AF6F960F3B2BF3AB), S3 Extended Request ID: CLs9xY8HAxbEAKEJC4LS1SgpqDcnHeaGocAbdsmYKwGttS64oVjFXJOe314vmb9q], ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], AWSRequestID=[AF6F960F3B2BF3AB], ServiceEndpoint=[https://kirk.s3.amazonaws.com], Exception=1, HttpClientPoolLeasedCount=0, RequestCount=1, HttpClientPoolPendingCount=0, HttpClientPoolAvailableCount=1, ClientExecuteTime=[10.111], HttpRequestTime=[10.009], HttpClientReceiveResponseTime=[8.758], RequestSigningTime=[0.043], HttpClientSendRequestTime=[0.044],
2015-09-04 20:48:19,685 INFO  [main] amazonaws.latency (AWSRequestMetricsFullSupport.java:log(203)) - StatusCode=[404], Exception=[com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: F2198ACEB4B2CE72), S3 Extended Request ID: J9oWD8ncn6WgfUhHA1yqrBfzFC+N533oD/DK90eiSvQrpGH4OJUc3riG2R4oS1NU], ServiceName=[Amazon S3], AWSErrorCode=[404 Not Found], AWSRequestID=[F2198ACEB4B2CE72], ServiceEndpoint=[https://kirk.s3.amazonaws.com], Exception=1, HttpClientPoolLeasedCount=0, RequestCount=1, HttpClientPoolPendingCount=0, HttpClientPoolAvailableCount=1, ClientExecuteTime=[9.879], HttpRequestTime=[9.776], HttpClientReceiveResponseTime=[8.537], RequestSigningTime=[0.05], HttpClientSendRequestTime=[0.033],

我可以理解是否有些用户对记录 S3 操作的延迟感兴趣，但是有什么方法可以禁用 AWSRequestMetricsFullSupport 的所有监控和日志记录吗？

当我检查 Spark UI 时，它告诉我作业完成得相对较快，但控制台在很长一段时间内充斥着这些消息。

Answer 1

响应。 AWS SDK for Java source comment阅读:

/**
 * Start an event which will be timed. [...]
 * 
 * This feature is enabled if the system property
 * "com.amazonaws.sdk.enableRuntimeProfiling" is set, or if a
 * {@link RequestMetricCollector} is in use either at the request, web service
 * client, or AWS SDK level.
 * 
 * @param eventName
 *            - The name of the event to start
 * 
 * @see AwsSdkMetrics
 */

如所引用的 AwsSdkMetrics Java Docs 中进一步概述，您可以通过系统属性禁用它:

The default metric collection of the Java AWS SDK is disabled by default. To enable it, simply specify the system property "com.amazonaws.sdk.enableDefaultMetrics" when starting up the JVM. When the system property is specified, a default metric collector will be started at the AWS SDK level. The default implementation uploads the request/response metrics captured to Amazon CloudWatch using AWS credentials obtained via the DefaultAWSCredentialsProviderChain.

这似乎可以被 RequestMetricCollector 硬连接在请求、Web 服务客户端或 AWS SDK 级别覆盖，这可能需要响应。正在使用的客户端/框架的配置调整(例如此处的 Spark):

Clients who needs to fully customize the metric collection can implement the SPI MetricCollector, and then replace the default AWS SDK implementation of the collector via setMetricCollector(MetricCollector).

到目前为止，这些功能的文档似乎有点稀疏，这里有两篇我知道的相关博客文章:

• Enabling Metrics with the AWS SDK for Java
• Metric Configuration in AWS SDK for Java