python - Tensorflow 的 Between-graph replication 是数据并行的一个例子吗？

Question

我已阅读 distributed tensorflow documentation和 this answer .

根据 this , 在数据并行方法中:

• The algorithm distributes the data between various cores.
• Each core independently tries to estimate the same parameter(s)
• Cores then exchange their estimate(s) with each other to come up with the right estimate for the step.

在模型并行方法中:

• The algorithm sends the same data to all the cores.
• Each core is responsible for estimating different parameter(s)
• Cores then exchange their estimate(s) with each other to come up with the right estimate for all the parameters.

In-graph replication 和Between-graph replication 与这些方法有什么关系？

This article说:

For example, different layers in a network may be trained in parallel on different GPUs. This training procedure is commonly known as "model parallelism" (or "in-graph replication" in the TensorFlow documentation).

并且:

In "data parallelism" (or “between-graph replication” in the TensorFlow documentation), you use the same model for every device, but train the model in each device using different training samples.

这准确吗？

来自Tensorflow DevSummit video链接在 tensorflow 文档页面: 看起来数据被拆分并分发给每个工作人员。那么 In-graph replication 不是遵循数据并行方法吗？

Answer 1

图内复制和图间复制与数据并行性和模型并行性没有直接关系。数据并行性和模型并行性是将并行化算法分为两类的术语，如您链接的 quora 答案中所述。但是图内复制和图间复制是tensorflow中实现并行的两种方式。例如，数据并行性可以通过图内复制和图间复制来实现。

如视频中所示，图内复制是通过将单个图的不同部分分配给不同的设备来实现的。在图间复制中，多个图并行运行，这是通过使用分布式 tensorflow 实现的。