ruby-on-rails - 如何使用事件记录批量处理 each_cons

Question

下面抛出一个LocalJumpError

records.find_each.each_cons(3)

是否有内置方法在内存友好的批处理中使用 each_cons？

---

编辑:

理想情况下，重叠会跨越批处理限制。 each_cons 在重叠组上进行交互，因此记录 ID 为 (0, 1, 2), (1, 2, 3), (2, 3, 4) 等。

如果批处理大小为 1000(我认为它适用于 Rails)，则将重叠限制在批处理组内并不是最佳选择。例如，索引为 (997, 998, 999) 的记录然后下一次迭代到 (1000, 1001, 1002) 是不可取的。

Answer 1

这是一个自定义方法来完成它:

def batched_each_cons(batch_size:, each_cons:, model:)
  end_of_previous = []
  model.find_in_batches(batch_size: batch_size) do |batch|
    rebuilt_batch = end_of_previous + batch
    end_of_previous = batch[(-(each_cons-1))..-1]
    rebuilt_batch.each_cons(each_cons) do |cons|
      yield cons
    end
  end
end

batched_each_cons(batch_size: 5, each_cons: 3, model: User) do | users |
  # puts users.map(&:name)
  puts users.map(&:id).inspect
end

这给出:

 User Load (0.7ms)  SELECT "users".* FROM "users" ORDER BY "users"."id" ASC LIMIT $1  [["LIMIT", 5]]
[1, 2, 3]
[2, 3, 4]
[3, 4, 5]
  User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."id" > $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 5], ["LIMIT", 5]]
[4, 5, 6]
[5, 6, 7]
[6, 7, 8]
[7, 8, 9]
[8, 9, 10]
  User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."id" > $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 10], ["LIMIT", 5]]
[9, 10, 11]
[10, 11, 12]
[11, 12, 13]
[12, 13, 14]
[13, 14, 15]
  User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."id" > $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 15], ["LIMIT", 5]]
[14, 15, 16]
[15, 16, 17]
[16, 17, 18]
[17, 18, 19]
[18, 19, 20]
  User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."id" > $1 ORDER BY "users"."id" ASC LIMIT $2  [["id", 20], ["LIMIT", 5]]

(这是在我的本地控制台中，通常您不会看到“User Load...”，这是表明批处理正在工作)