ruby-on-rails - 在未调用的辅助方法中打开 CSV 文件

Question

我需要读取存储在 S3 上的 CSV，并将其保存到 Heroku 上的 postgresql 数据库中。

在开发中，这工作正常，但文件保存在本地而不是

CSV.foreach(file.path, :headers => true, skip_blanks: true) do |row|
    puts 'CSV is open'
    puts row
    if row.to_hash.values.all?(&:nil?)
    else
        @datum = Datum.new
        @datum.attributes = row.to_hash.reject{|k,v| !@datum.attributes.keys.member?(k.to_s) }
        @datum.user_id = current_user_id
        @datum.batch_id = batch_id
        @datum.save
    end
end

我找到了 this post ，并将 require 'open-uri' 和 require 'csv' 添加到 /app/workers/batch_worker.rb

然后我改成了

 CSV.new(open(file), :headers => true, skip_blanks: true) do |row|
      puts 'CSV is open'
      puts row
      if row.to_hash.values.all?(&:nil?)
      else
        @datum = Datum.new
        @datum.attributes = row.to_hash.reject{|k,v| !@datum.attributes.keys.member?(k.to_s) }
        @datum.user_id = current_user_id
        @datum.batch_id = batch_id
        @datum.save
      end
 end

batches_worker.rb 中的完整方法:

 def perform(service_id, batch_id, current_user_id, file)
    batch = Batch.find(batch_id)

    CSV.new(open(file), :headers => true, skip_blanks: true) do |row|
      puts 'CSV is open'
      puts row
      if row.to_hash.values.all?(&:nil?)
      else
        @datum = Datum.new
        @datum.attributes = row.to_hash.reject{|k,v| !@datum.attributes.keys.member?(k.to_s) }
        @datum.user_id = current_user_id
        @datum.batch_id = batch_id
        @datum.save
      end
    end

    if service_id.to_i === 3
        batch.progress = "Verify Email In Progress"
        batch.save
        verify_email(batch_id, current_user_id)
    elsif service_id.to_i === 5
        batch.progress = "Verify Phone In Progress"
        batch.save
        verify_active_phone(batch_id, current_user_id)
    elsif service_id.to_i === 1
        batch.progress = "Append Email In Progress"
        batch.save
        append_email(batch_id, current_user_id)
    elsif service_id.to_i === 4
        batch.progress = "Append Phone In Progress"
        batch.save
        append_phone(batch_id, current_user_id)
     elsif service_id.to_i === 2
        batch.progress = "Append Name In Progress"
        batch.save
        append_name_address(batch_id, current_user_id)
    end
end

我的服务器日志。没有错误，但它完全超过了整个 CSV 部分。

2018-01-30T01:24:37.112667+00:00 app[worker.2]: 4 TID-vco0s BatchWorker JID-b0a151240278a0063f608da7 INFO: start
2018-01-30T01:24:37.116144+00:00 app[worker.2]:   Batch Load (1.4ms)  SELECT  "batches".* FROM "batches" WHERE "batches"."id" = $1 LIMIT $2  [["id", 55], ["LIMIT", 1]]
2018-01-30T01:24:37.116377+00:00 app[worker.2]: file is https://s3.amazonaws.com/example/example/csv_files/000/000/055/original/brick.csv
2018-01-30T01:24:37.147023+00:00 app[worker.2]:    (1.1ms)  BEGIN
2018-01-30T01:24:37.149477+00:00 app[worker.2]:   User Load (1.2ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT $2  [["id", 1], ["LIMIT", 1]]
2018-01-30T01:24:37.152710+00:00 app[worker.2]:   SQL (1.3ms)  UPDATE "batches" SET "progress" = $1, "updated_at" = $2 WHERE "batches"."id" = $3  [["progress", "Verify Email In Progress"], ["updated_at", "2018-01-30 01:24:37.150315"], ["id", 55]]

我更改了帖子的存储桶名称，但如果我到达 url(放入日志)，它会从 aws 下载 CSV - 所以我知道链接是正确的(事实上没有错误)。好像是CSV

我已经研究了几个小时 - 希望一些新鲜的眼光能看到这个问题。谢谢!

Answer 1

万一其他人遇到这个问题，我认为问题是在工作人员中，它正在接收一个字符串，而在 Rails Controller 中，它理解它是一个 URL 并且需要通过 csv 发送...我我不确定为什么它没有抛出错误。

这是我的工作代码:

class BatchWorker
   include Sidekiq::Worker
   require 'open-uri'
   require 'net/https'
   require 'csv'

  def perform(service_id, batch_id, current_user_id, file)
    batch = Batch.find(batch_id)

    escaped_link = URI.escape(file) 
    file = URI.parse(escaped_link)
    puts Net::HTTP.get(file)

    CSV.parse(Net::HTTP.get(file), :headers => true, skip_blanks: true) do |row|
      puts 'CSV is open'
      puts row
      if row.to_hash.values.all?(&:nil?)
      else
        @datum = Datum.new
        @datum.attributes = row.to_hash.reject{|k,v| !@datum.attributes.keys.member?(k.to_s) }
        @datum.user_id = current_user_id
        @datum.batch_id = batch_id
        @datum.save
      end
    end
  end
end