java - 我如何在 Java 中同时完成 80,000 次下载？

Question

我正在克隆大量 GitHub 项目样本以进行实证研究。我假设下载具有一定并发性的 80,000 个项目会更快，但要下载的东西太多了。

如何启动大约 1,000 个进程，然后在每个进程完成后启动另一个进程？或者，我还有其他方法可以解决这个问题吗？以比连续速度更快的速度下载这么多会对 GitHub 的服务器不利吗？

到目前为止，这是相关代码:

// Create a CountDownLatch that will only reach 0 when all repositories
// have been downloaded
CountDownLatch doneSignal = new CountDownLatch(numberOfRepositories);

// Start the download for each git repository
for (String URL : gitURLs)
{
    new Thread(new Worker(doneSignal, URL)).start();
}

doneSignal.await();

worker :

public class Worker implements Runnable
{
    private final CountDownLatch doneSignal;
    private final String URL;

    Worker (CountDownLatch doneSignal, String URL)
    {
        this.doneSignal = doneSignal;
        this.URL = URL;
    }

    @Override
    public void run ()
    {
        try
        {
            // Run the command line process to download
            ProcessBuilder pb =
                new ProcessBuilder("git", "clone", "--depth=1", URL, "projects/" + getProjectName(URL));
            Process p = pb.start();
            p.waitFor();
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }

        doneSignal.countDown();
    }
}

Answer 1

这对 github 的服务器不利，但对您的性能更不利。尝试使用 5 个左右而不是 1000 个。要将代码限制为 X 个并行线程，您可以使用池:

CountDownLatch doneSignal = new CountDownLatch(numberOfRepositories);
// Start the download for each git repository
ExecutorService pool = Executors.newFixedThreadPool(5);
for (String URL : gitURLs) {
    pool.execute(new Worker(doneSignal, URL));
}
pool.shutdown();
doneSignal.await();

也可以在没有闩锁的情况下工作，因为您可以通过例如等待池空闲

pool.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);