git - 如何使用 git clone --recursive 加速/并行化 git 子模块的下载？

Question

克隆具有大量子模块的 git 存储库需要很长时间。在下面的例子中是 ~100 个子模块

git clone --recursive https://github.com/Whonix/Whonix

Git 将它们一一克隆。花费的时间比要求的要长得多。让我们(可能)假设客户端和服务器都有足够的资源来同时响应多个(并行)请求。

如何使用 git clone --recursive 加速/并行下载 git 子模块？

Answer 1

使用 git 2.8 (Q12016)，您将能够启动子模块的获取...并行!

参见 commit fbf7164 (2015 年 12 月 16 日)Jonathan Nieder ( artagnon ) .
参见 commit 62104ba , commit fe85ee6 , commit c553c72 , commit bfb6b53 , commit b4e04fb , commit 1079c4b (2015 年 12 月 16 日)Stefan Beller ( stefanbeller ) .
^{(由 Junio C Hamano -- gitster -- merge 于 commit 187c0d3 ，2016 年 1 月 12 日)}

Add a framework to spawn a group of processes in parallel, and use it to run "git fetch --recurse-submodules" in parallel.

为此，git fetch有新选项:

-j, --jobs=<n>

Number of parallel children to be used for fetching submodules.
Each will fetch from different submodules, such that fetching many submodules will be faster.
By default submodules will be fetched one at a time.

例子:

git fetch --recurse-submodules -j2

---

大部分新功能都在 commit c553c72 中(2015 年 12 月 16 日)Stefan Beller ( stefanbeller ) .

run-command: add an asynchronous parallel child processor

This allows to run external commands in parallel with ordered output on stderr.

If we run external commands in parallel we cannot pipe the output directly to the our stdout/err as it would mix up. So each process's output will flow through a pipe, which we buffer. One subprocess can be directly piped to out stdout/err for a low latency feedback to the user.

---

请注意，在 Git 2.24(2019 年第 4 季度)之前，“git fetch --jobs=<n>”允许​​ <n>获取子模块时的并行作业，但这不适用于从多个远程存储库获取的“git fetch --multiple”。
现在可以了。

参见 commit d54dea7 (2019 年 10 月 5 日)作者:Johannes Schindelin ( dscho ) .
^{(由 Junio C Hamano -- gitster -- merge 于 commit d96e31e ，2019 年 10 月 15 日)}

fetch: let --jobs=<n> parallelize --multiple, too

^{Signed-off-by: Johannes Schindelin}

So far, --jobs=<n> only parallelizes submodule fetches/clones, not --multiple fetches, which is unintuitive, given that the option's name does not say anything about submodules in particular.

Let's change that.
With this patch, also fetches from multiple remotes are parallelized.

For backwards-compatibility (and to prepare for a use case where submodule and multiple-remote fetches may need different parallelization limits):

• the config setting submodule.fetchJobs still only controls the submodule part of git fetch,
• while the newly-introduced setting fetch.parallel controls both (but can be overridden for submodules with submodule.fetchJobs).