gpt4 book ai didi

c++ - 上传图像文件在 AWS S3 C++ SDK 上花费的时间太长

转载 作者:太空宇宙 更新时间:2023-11-04 12:34:30 24 4
gpt4 key购买 nike

使用 AWS S3 C++ SDK 将 .jpg 图像上传到某个 IAM 用户会引入巨大的时间延迟,这在任何情况下都是由于网络流量和延迟问题引起的。我为我的应用程序(在 Windows 10 PC 上)使用免费层 S3 版本和 MSVC 2017 64 位。这是一个示例代码:

Aws::SDKOptions options;
Aws::InitAPI(options);

Aws::Client::ClientConfiguration config;
config.region = Aws::Region::US_EAST_2;

Aws::S3::S3Client s3_client(Aws::Auth::AWSCredentials(KEY,ACCESS_KEY), config);

const Aws::String bucket_name = BUCKET;
const Aws::String object_name = "image.jpg";

Aws::S3::Model::PutObjectRequest put_object_request;

put_object_request.SetBucket(bucket_name);
put_object_request.SetKey(object_name);

std::shared_ptr<Aws::IOStream> input_data =
Aws::MakeShared<Aws::FStream>("PutObjectInputStream",
"../image.jpg",
std::ios_base::in | std::ios::binary);

put_object_request.SetBody(input_data);
put_object_request.SetContentType("image/jpeg");

input_data->seekg(0LL, input_data->end);
put_object_request.SetContentLength(static_cast<long>(input_data->tellg()));

auto put_object_outcome = s3_client.PutObject(put_object_request);

当我上传的图片总大小超过 100KB

PutObject(put_object_request);

对于 520KB 的图像,执行时间超过 2 分钟。

我使用 Python boto3 尝试了相同的示例,同一图像的总上传时间约为 25 秒。

有人遇到过同样的问题吗?

最佳答案

更好看后去AWS github repo我弄清楚了这个问题。

问题是 WinHttpSyncHttpClient 正在超时并在内部 重置上传事件,因此不会退出上传线程并最终中止事务。通过添加自定义超时值,问题就解决了。

我使用分段上传来重新实现该示例,因为它看起来更健壮且更易于管理。虽然我认为它不适用于 C++ SDK,但事实并非如此 TransferManager对 C++ 做同样的工作(不像 Java、.NET 和 PHP 那样使用 S3 header )。

谢谢 KaibaLopezSCalwas来自帮助我解决问题的 AWS github repo(issue1issue2)。我正在粘贴一个示例代码,以防有人遇到同样的问题:

#include "pch.h"
#include <iostream>
#include <fstream>
#include <filesystem>

#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentials.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/Bucket.h>
#include <aws/transfer/TransferManager.h>
#include <aws/transfer/TransferHandle.h>

static const char* KEY = "KEY";
static const char* BUCKET = "BUCKET_NAME";
static const char* ACCESS_KEY = "AKEY";
static const char* OBJ_NAME = "img.jpg";
static const char* const ALLOCATION_TAG = "S3_SINGLE_OBJ_TEST";

int main()
{
Aws::SDKOptions options;
Aws::InitAPI(options);
{
Aws::Client::ClientConfiguration config;
config.region = Aws::Region::US_EAST_2;
config.requestTimeoutMs = 20000;

auto s3_client = std::make_shared<Aws::S3::S3Client>(Aws::Auth::AWSCredentials(KEY, ACCESS_KEY), config);

const Aws::String bucket_name = BUCKET;
const Aws::String object_name = OBJ_NAME;
const Aws::String key_name = OBJ_NAME;

auto s3_client_executor = Aws::MakeShared<Aws::Utils::Threading::DefaultExecutor>(ALLOCATION_TAG);
Aws::Transfer::TransferManagerConfiguration trConfig(s3_client_executor.get());
trConfig.s3Client = s3_client;


trConfig.uploadProgressCallback =
[](const Aws::Transfer::TransferManager*, const std::shared_ptr<const Aws::Transfer::TransferHandle>&transferHandle)
{ std::cout << "Upload Progress: " << transferHandle->GetBytesTransferred() <<
" of " << transferHandle->GetBytesTotalSize() << " bytes" << std::endl;};

std::cout << "File start upload" << std::endl;

auto tranfer_manager = Aws::Transfer::TransferManager::Create(trConfig);
auto transferHandle = tranfer_manager->UploadFile(object_name.c_str(),
bucket_name.c_str(), key_name.c_str(), "multipart/form-data", Aws::Map<Aws::String, Aws::String>());

transferHandle->WaitUntilFinished();

if(transferHandle->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED)
std::cout << "File up" << std::endl;
else
std::cout << "Error uploading: " << transferHandle->GetLastError() << std::endl;
}
Aws::ShutdownAPI(options);
return 0;
}

关于c++ - 上传图像文件在 AWS S3 C++ SDK 上花费的时间太长,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56989720/

24 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com