amazon-web-services - AWS Kinesis 如何限制写入吞吐量？

Question

AWS Kinesis 的写入吞吐量相当低，为 1000 次写入/秒和 1MB/写入-秒。 Kinesis 如何执行此限制？如果我尝试在一秒钟内进行 1500 次写入，那么额外的 500 次写入会被放入某种队列中，还是会失败？

Answer 1

看起来它只是失败并引发了异常。

An unsuccessfully processed record includes ErrorCode and ErrorMessage values. ErrorCode reflects the type of error and can be one of the following values: ProvisionedThroughputExceededException or InternalFailure. ErrorMessage provides more detailed information about the ProvisionedThroughputExceededException exception including the account ID, stream name, and shard ID of the record that was throttled. For more information about partially successful responses, see Adding Multiple Records with PutRecords in the Amazon Kinesis Data Streams Developer Guide.

https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html

限速是如何完成的

Rate Limiting The KPL includes a rate limiting feature, which limits per-shard throughput sent from a single producer. Rate limiting is implemented using a token bucket algorithm with separate buckets for both Kinesis Data Streams records and bytes. Each successful write to an Kinesis data stream adds a token (or multiple tokens) to each bucket, up to a certain threshold. This threshold is configurable but by default is set 50% higher than the actual shard limit, to allow shard saturation from a single producer.

You can lower this limit to reduce spamming due to excessive retries. However, the best practice is for each producer is to retry for maximum throughput aggressively and to handle any resulting throttling determined as excessive by expanding the capacity of the stream and implementing an appropriate partition key strategy.

https://docs.aws.amazon.com/streams/latest/dev/kinesis-producer-adv-retries-rate-limiting.html