artificial-intelligence - 人工智能: How would I train a Neural Network across multiple machines?

Question

因此，对于具有大量数据集的大型网络，它们需要一段时间来训练。如果有一种方法可以在多台机器之间共享计算时间，那就太棒了。然而，问题在于，当神经网络在训练时，每次迭代的权重都在不断地改变，每次迭代或多或少都基于最后一次——这使得分布式计算的想法至少是一个挑战.

我认为对于网络的每个部分，服务器可能会发送 1000 组数据来训练网络……但是……您的计算时间与我无法实现的计算时间大致相同同时训练不同的数据集(这是我想要做的)。

但即使我可以将网络的训练分成不同的数据集块进行训练，我怎么知道我何时完成了这组数据？特别是如果发送到客户端计算机的数据量不足以实现所需的错误？

我欢迎所有想法。

Answer 1

报价http://en.wikipedia.org/wiki/Backpropagation#Multithreaded_Backpropagation :

When multicore computers are used multithreaded techniques can greatly decrease the amount of time that backpropagation takes to converge. If batching is being used, it is relatively simple to adapt the backpropagation algorithm to operate in a multithreaded manner.

The training data is broken up into equally large batches for each of the threads. Each thread executes the forward and backward propagations. The weight and threshold deltas are summed for each of the threads. At the end of each iteration all threads must pause briefly for the weight and threshold deltas to be summed and applied to the neural network.

这基本上是这里其他答案所描述的。