ruby-on-rails - rails : Uploading a file to AS3 using fog and resque

Question

问题:

我有一个 Rails 应用程序，它要求用户上传某种类型的电子表格(csv、xslx、xsl 等)进行处理，这可能是一项成本高昂的操作，因此我们决定将其发送到后台服务作为解决方案到这个问题。我们关心的问题是，因为我们的生产系统是在 Heroku 上，所以我们需要先将文件存储在 AS3 上，然后再检索进行处理。

因为将文件上传到 AS3 本身就是一项成本高昂的操作，所以这也应该作为后台作业来完成。问题在于，由于 Resque 需要将文件数据放入 Redis 或稍后检索，因此使用 Resque 执行此操作可能会占用大量 RAM。如您所知，Redis 仅将其数据存储在 RAM 中，并且更喜欢简单的键值对，因此我们希望尽量避免这种情况。

下面是一些伪代码，作为我们想要尝试做的示例:

workers/AS3Uploader.rb

require 'fog'

class AS3Uploader
  @queue = :as3_uploader
  def self.perform(some, file, data)
    # create a connection
    connection = Fog::Storage.new({
      :provider                 => 'AWS',
      :aws_access_key_id        => APP_CONFIG['s3_key'],
      :aws_secret_access_key    => APP_CONFIG['s3_secret']
    })

    # First, a place to contain the glorious details
    directory = connection.directories.create(
      :key    => "catalog-#{Time.now.to_i}", # globally unique name
      :public => true
    )

    # list directories
    p connection.directories

    # upload that catalog
    file = directory.files.create(
      :key    => 'catalog.xml',
      :body   => File.open(blah), # not sure how to get file data here with out putting it into RAM first using Resque/Redis
      :public => true
  end

  # make a call to Enqueue the processing of the catalog
  Resque.enqueue(CatalogProcessor, some, parameters, here)
end

controllers/catalog_upload_controller.rb

def create
  # process params

  # call Enqueue to start the file processing
  # What do I do here? I could send all of the file data here right now
  # but like I said previously that means storing potentially 100s of MB into RAM
  Resque.enqueue(AS3Uploader, some, parameters, here)
end

Answer 1

我建议你做的方式是

• 将您的文件存储在您创建的 tmp 目录中并获取 file-path
• 告诉 Resque 使用 file-path 上传文件
• 制作 Resque 将 file-path 存储在 redis 而不是整个 file-content (它会很贵)
• 现在工作人员将文件上传到AWS-S3

Note: If you have multiple instances like One instance for background processing, One for database, One as utility instance then your tmp dir may not be available to other instances.. so store the file in the temp dir inside the instance holding the resque