ruby-on-rails - 如何同步新的 ActiveStorage 镜像？

Question

从 ActiveStorage 开始，您可以知道定义用于存储文件的镜像。

local:
  service: Disk
  root: <%= Rails.root.join("storage") %>

amazon:
  service: S3
  access_key_id: <%= Rails.application.credentials.dig(:aws, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:aws, :secret_access_key) %>
  region: us-east-1
  bucket: mybucket

mirror:
  service: Mirror
  primary: local
  mirrors:
    - amazon
    - another_mirror

如果在某个时间点后添加镜像，则必须注意复制所有文件，例如从“本地”到“亚马逊”或“another_mirror”。

有没有方便的方法来保持文件同步？

或者方法运行验证以检查每个服务上的所有文件是否可用？

Answer 1

我有几个可能适合您的解决方案，一个适用于 Rails <= 6.0，另一个适用于 Rails >= 6.1:
首先，您需要遍历 ActiveStorage Blob:

ActiveStorage::Blob.all.each do |blob|
  # work with blob
end

然后...

rails <= 6.0
您将需要 blob 的 key 、校验和以及磁盘上的本地文件。

local_file = ActiveStorage::Blob.service.primary.path_for blob.key

# I'm picking the first mirror as an example,
# but you can select a specific mirror if you want
mirror = blob.service.mirrors.first

mirror.upload blob.key, File.open(local_file), checksum: blob.checksum

如果镜像已存在，您可能还想避免上传文件。你可以这样做:

mirror = blob.service.mirrors.first

# If the file doesn't exist on the mirror, upload it
unless mirror.exist? blob.key
  # Upload file to mirror
end

把它放在一起，一个 rake 任务可能看起来像:

# lib/tasks/active_storage.rake

namespace :active_storage do

  desc 'Ensures all files are mirrored'
  task mirror_all: [:environment] do

  # Iterate through each blob
  ActiveStorage::Blob.all.each do |blob|

    # We assume the primary storage is local
    local_file = ActiveStorage::Blob.service.primary.path_for blob.key

    # Iterate through each mirror
    blob.service.mirrors.each do |mirror|

      # If the file doesn't exist on the mirror, upload it
      mirror.upload(blob.key, File.open(local_file), checksum: blob.checksum) unless mirror.exist? blob.key

      end
    end
  end
end

您可能会遇到类似 @Rystraum mentioned 的情况。您可能需要从本地磁盘以外的其他地方进行镜像。在这种情况下，rake 任务可能如下所示:

# lib/tasks/active_storage.rake

namespace :active_storage do

  desc 'Ensures all files are mirrored'
  task mirror_all: [:environment] do

    # All services in our rails configuration
    all_services = [ActiveStorage::Blob.service.primary, *ActiveStorage::Blob.service.mirrors]

    # Iterate through each blob
    ActiveStorage::Blob.all.each do |blob|

      # Select services where file exists
      services = all_services.select { |file| file.exist? blob.key }

      # Skip blob if file doesn't exist anywhere
      next unless services.present?

      # Select services where file doesn't exist
      mirrors = all_services - services

      # Open the local file (if one exists)
      local_file = File.open(services.find{ |service| service.is_a? ActiveStorage::Service::DiskService }.path_for blob.key) if services.select{ |service| service.is_a? ActiveStorage::Service::DiskService }.any?

      # Upload local file to mirrors (if one exists)
      mirrors.each do |mirror|
        mirror.upload blob.key, local_file, checksum: blob.checksum
      end if local_file.present?

      # If no local file exists then download a remote file and upload it to the mirrors (thanks @Rystraum)
      services.first.open blob.key, checksum: blob.checksum do |temp_file|
        mirrors.each do |mirror|
          mirror.upload blob.key, temp_file, checksum: blob.checksum
        end
      end unless local_file.present?

    end
  end
end

虽然第一个 rake 任务回答了 OP 的问题，但后者更加通用:

它可以与任何服务组合一起使用

不需要 DiskService

优先通过 DiskServices 上传

避免额外的存在？调用，因为我们每个 blob 每个服务只调用一次

rails > 6.1
它 super 简单，只需在每个 blob 上调用它...

blob.mirror_later

将其包装为 rake 任务如下所示:

# lib/tasks/active_storage.rake

namespace :active_storage do

  desc 'Ensures all files are mirrored'
  task mirror_all: [:environment] do
    ActiveStorage::Blob.all.each do |blob|
      blob.mirror_later
    end
  end
end