amazon-s3 - 如何使用 AWS S3 put 对象解决减速错误

Question

我正在尝试访问一个文件并使用 boto 在 S3 中更新它，但即使按照下面的代码在请求之间暂停后，仍然会继续出现减速错误。我该如何解决这个问题？

body = b'Here we have some more data'
s3.put_object(Body=body,Bucket=bucket, Key=key)
time.sleep(10)
response = s3.get_object(Bucket=bucket, Key=key)
time.sleep(10)
print(response["Body"].read().decode('utf-8'))
currFile = response["Body"].read().decode('utf-8')
newFile = currFile + "\n" + "New Stuff!!!"
newFileB = newFile.encode('utf-8')
time.sleep(60)
s3.put_object(Body=newFileB,Bucket=bucket, Key=key)
time.sleep(10)
response = s3.get_object(Bucket=bucket, Key=key)
print(response["Body"].read().decode('utf-8'))

这是错误:

Details
The area below shows the result returned by your function execution.
{
"errorMessage": "An error occurred (SlowDown) when calling the PutObject operation (reached max retries: 4): Please reduce your request rate.",
"errorType": "ClientError",
"stackTrace": [
[
"/var/task/lambda_function.py",
43,
"lambda_handler",
"raise e"
],
[
"/var/task/lambda_function.py",
20,
"lambda_handler",
"s3.put_object(Body=body,Bucket=bucket, Key=key)"
],
[
"/var/runtime/botocore/client.py",
314,
"_api_call",
"return self._make_api_call(operation_name, kwargs)"
],
[
"/var/runtime/botocore/client.py",
612,
"_make_api_call",
"raise error_class(parsed_response, operation_name)"
]
]
}

Answer 1

我遇到了这个确切的问题，我不确定它为什么会发生，但这是生产代码必须一直处理的事情。解决的办法是坚持尝试，不要放弃很长一段时间。失败时，循环以 1 秒的延迟开始，然后在每个循环中将延迟增加 1 秒 (delay_incr)，最后每个循环的 max_delay 超时 30 秒，实际上最多总共 7.5 分钟，当它终于放弃了。当然，你可以调整时间。到目前为止，这对我来说是成功的。
即使对于有时我必须等待一段时间才能读取文件的 NAS 文件服务器，我也必须做类似的事情。

def put_s3_core(bucket, key, strobj, content_type=None):  
    """ write strobj to s3 bucket, key
        content_type can be:
            binary/octet-stream (default)
            text/plain
            text/html
            text/csv
            image/png
            image/tiff
            application/pdf
            application/zip
            
    """
    delay = 1       # initial delay
    delay_incr = 1  # additional delay in each loop
    max_delay = 30  # max delay of one loop. Total delay is (max_delay**2)/2
    
    while delay < max_delay:   
        try:
            s3 = boto3.resource('s3')
            request_obj = s3.Object(bucket, key)
            if content_type:
                request_obj.put(Body=strobj, ContentType=content_type)        
            else:
                request_obj.put(Body=strobj)
            break               
                
        except ClientError:
            time.sleep(delay)
            delay += delay_incr
    else:
        raise
    ```