ruby-on-rails - 在使用线程时处理 Rails 中的循环依赖

Question

这里有五个文件添加到 my app (没有数据库，不需要这样的设置):

lib/tasks/precomputation.rake

namespace :precomputation do
  desc "This fetches data for precomputation"
  task fetch_all: :environment do
      Precomputation.precompute_all_data
    # end
  end
end

应用程序/模型/precomputation.rb

class Precomputation
  def self.precompute_all_data
    ad_accounts = [1,2,3]
    bgthread = BackgroundThread::BGThreadPool.new(1)
    tasks = []
    ad_accounts.each do |ad_account_id|
      p = Proc.new do
        begin
          MongoPipeline::FbAdCampaignMongoPipeline.new(ad_account_id).fetch_all
          false
        ensure
          GC.start
        end
      end
      tasks << [p, "test #{ad_account_id}"]
    end
    bgthread.add_randomized_tasks(tasks)
    bgthread.do_work
  end
end

应用程序/模型/mongo_pipeline.rb

module MongoPipeline
  class Base
    def initialize(ad_account_id)
    end

    def insert_data
      puts 'inserting data'
    end

    def fetch_all
      extract_data # Child Class defines this method
      insert_data # Base class defines this method
    end
  end
end

app/models/mongo_pipeline/fb_ad_campaign_mongo_pipeline.rb

module MongoPipeline
  class FbAdCampaignMongoPipeline < MongoPipeline::Base
    def extract_data
      puts 'here is campaign data'
    end
  end
end

和 app/models/background_thread.rb

(注意:使用并行 gem 且无后台线程的替代实现也会因类似错误而崩溃 -:https://github.com/pratik60/circular_dependency_havoc/tree/parallel)

错误日志

rake aborted!
Circular dependency detected while autoloading constant MongoPipeline
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:492:in `load_missing_constant'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:184:in `const_missing'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:526:in `load_missing_constant'
/Users/pratikbothra/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.4/lib/active_support/dependencies.rb:184:in `const_missing'
/webapps/circular_dependency_havoc/app/models/precomputation.rb:9:in `block (2 levels) in precompute_all_data'
/webapps/circular_dependency_havoc/app/models/background_thread.rb:91:in `call'
/webapps/circular_dependency_havoc/app/models/background_thread.rb:91:in `block in background'
Tasks: TOP => precomputation:fetch_all
(See full trace by running task with --trace)

关于我做错了什么的任何想法？克隆了后台线程库，只是修改了一下。如果您认为这是问题所在，请随时更换它。我们非常欢迎任何建议和想法。

Answer 1

我在 rake 任务中构建线程循环以轮询外部服务器时遇到了这个问题，并得出了相同的结论，即它与自动加载和不可预测的竞争条件相关。但是我不想简单地打开预加载，所以我有两种方法来解决这个问题。一种是在生成任何线程之前找出我的任务中加载了哪些类并将它们加载到父级中:

# preload classes that showed circular dependencies just in this task
MediaFile
VideoPlatformIntegration
Tag

10.times.map{Thread.new{...}}

这就像一个魅力，但放入生产代码中有点奇怪，并且对任务中的任何更改或它调用的任何方法都不健壮。

另一个更不完美，但实际上只是修改了开发环境中的行为，并不依赖于了解可能导致这种情况的一切。由于它只是在开发中，我认为这是可以接受的:在每个初始线程开始时随机休眠一段时间(0 到 1 秒之间):

如果 Rails.env == "development"，则 sleep (Random.rand)

这种休眠可以防止线程同时启动，并大大减少加载类时出现任何竞争条件的可能性。我在这里使用 10 个线程的简单批处理进行概念验证，在有 sleep 的情况下运行脚本 4 次，在没有 sleep 的情况下运行脚本 4 次，在没有 sleep 的情况下显示可靠的循环依赖错误:

[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
Circular dependency detected while autoloading constant MediaFile
Circular dependency detected while autoloading constant MediaFile
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
Circular dependency detected while autoloading constant MediaFile
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{MediaFile.last rescue puts($!)}}'
Circular dependency detected while autoloading constant MediaFile
Circular dependency detected while autoloading constant MediaFile
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'
[video_platform_integration] andrew@~/3p/app3$ rails runner '10.times.map{Thread.new{sleep(Random.rand); MediaFile.last rescue puts($!)}}'