ruby-on-rails - 503 使用 Carrierwave 上传到 S3 时速度变慢

Question

我在后台作业中使用 Rails 和 Carrierwave 将许多小文件上传到 S3，并且达到了 S3 速率限制。我的直接想法是放一个 sleep 0.1在每次上传之前，但这似乎不是一个很好的解决方案。
关于如何通过 S3 API 和某种类型的退避来处理这个问题的任何建议？
正在执行上传的 Ruby 代码，此方法在循环中被调用数千次:

    def attach_audio(object:, audio_field:, attachment:)
      return true if Rails.env.test?

      language_code, voice_id = language_and_voice(object)

      resp = polly.synthesize_speech(
        output_format: 'mp3',
        voice_id: voice_id,
        text: audio_field.to_s,
        language_code: language_code
      )

      audio_filename = "#{object.class.to_s.downcase}_#{attachment}_#{object.id}_#{voice_id}.mp3"
      audio_path = "#{Rails.root}/db/audio/#{audio_filename}"
      IO.copy_stream(resp.audio_stream, audio_path)

      object.send(attachment + '=', Pathname.new(audio_path).open)
      object.save!
    end

上传类

class AudioUploader < BaseUploader

  def store_dir
    "uploads/audio/#{model.target_language}/#{self.class.to_s.underscore}/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

  def extension_whitelist
    %w[mp3]
  end
end

class BaseUploader < CarrierWave::Uploader::Base
  if Rails.env.test?
    storage :file
  else
    storage :fog
  end

  def store_dir
    "uploads/#{self.class.to_s.underscore}/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end
end

来自 AWS 的回应

Message

Excon::Error::ServiceUnavailable: Expected(200) <=> Actual(503 Service Unavailable) excon.error.response :body => "<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>176C22715A856A29</RequestId><HostId>L/+

Traceback

Excon::Error::ServiceUnavailable: Expected(200) <=> Actual(503 Service Unavailable)
excon.error.response
  :body          => "<Error><Code>SlowDown</Code><Message>Please reduce your request rate.</Message><RequestId>176C22715A856A29</RequestId><HostId>xxxxxxxxxxxxxxxxxxxxxxxxx</HostId></Error>"
  :cookies       => [
  ]
  :headers       => {
    "Connection"       => "close"
    "Content-Type"     => "application/xml"
    "Date"             => "Wed, 18 Nov 2020 07:31:29 GMT"
    "Server"           => "AmazonS3"
    "x-amz-id-2"       => "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    "x-amz-request-id" => "176C22715A856A29"
  }
  :host          => "example-production.s3-eu-west-1.amazonaws.com"
  :local_address => "xxx.xx.xxx.xxx"
  :local_port    => 50276
  :path          => "/uploads/audio/fr/audio_uploader/word/audio_file/8015423/word_audio_file_8015423_Mathieu.mp3"
  :port          => 443
  :reason_phrase => "Slow Down"
  :remote_ip     => "xx.xxx.xx.x"
  :status        => 503
  :status_line   => "HTTP/1.1 503 Slow Down\r\n"

  File "/app/vendor/bundle/ruby/2.6.0/gems/excon-0.71.1/lib/excon/middlewares/expects.rb", line 13, in response_call
  File "/app/vendor/bundle/ruby/2.6.0/gems/excon-0.71.1/lib/excon/middlewares/response_parser.rb", line 12, in response_call
  File "/app/vendor/bundle/ruby/2.6.0/gems/excon-0.71.1/lib/excon/connection.rb", line 448, in response
  File "/app/vendor/bundle/ruby/2.6.0/gems/excon-0.71.1/lib/excon/connection.rb", line 279, in request
  File "/app/vendor/bundle/ruby/2.6.0/gems/fog-xml-0.1.3/lib/fog/xml/sax_parser_connection.rb", line 35, in request

etc

编辑
链接的 AWS 文档指的是前缀，这似乎可以解决问题

Amazon S3 automatically scales to high request rates. For example, your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in a bucket. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by parallelizing reads. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second.

但我不明白如何在 Carrierwave 的上下文中实现它。

Answer 1

来自 here

For example, your application can achieve at least 3,500PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefixin a bucket.

你知道你的极限是什么。现在您需要了解什么是 前缀 是，这很容易。考虑一下:

/uploads/audio/fr/audio_uploader/word/audio_file/8015423/word_audio_file_8015423_Mathieu.mp3

什么是 前缀 这里？回答:

/uploads/audio/fr/audio_uploader/word/audio_file/8015423

前缀 是除对象名称之外的所有内容。因此，您的问题的答案在于您设计方案的能力，以便您永远不会超过亚马逊为每个 定义的限制。前缀 .
例如，您可以使用一个旋转计数器，比如说从 0 到 99，并将被保存的对象与存储它的旋转计数器位置之间的关系存储在某处[以便您以后阅读]。如果你实现这个，你的问题将减少到现在的 1/100；您实际上可能不需要一直到 100，如果需要，您可以在将来随时增加它。所以现在，这个:

/uploads/audio/fr/audio_uploader/word/audio_file/8015423/word_audio_file_8015423_Mathieu.mp3

将存储在:

/uploads/audio/fr/audio_uploader/word/audio_file/00/8015423/word_audio_file_8015423_Mathieu.mp3

下一个在 .../01/... 依此类推，第 100 个对象存储在 中。 .../99/... 然后将第 101 个对象存储回 .../00/... [您显然不必使用这两个字符]。
此过程为您的逻辑带来的额外步骤是，出于检索目的，您需要知道 word_audio_file_8015423_Mathieu.mp3 位于 中。 .../00/... 例如，word_audio_file_8015424_Mark.mp3 位于 .../01/... 等等。这意味着您必须存储对象与其保存位置之间的关系。另一方面，如果可以搜索所有点来寻找您想要的对象，您甚至可能不需要这样做。
我强烈认为这会解决您的问题。