c# - 在 for 循环中每 x 次运行异步

Question

我正在下载超过 10 万个文件，并希望以补丁的形式进行下载，例如一次下载 100 个文件。

static void Main(string[] args) {
    Task.WaitAll(
      new Task[]{
           RunAsync()
    });
}

// each group has 100 attachments.
static async Task RunAsync() {
    foreach (var group in groups) {
        var tasks = new List<Task>();
        foreach (var attachment in group.attachments) {
            tasks.Add(DownloadFileAsync(attachment, downloadPath));
        }
        await Task.WhenAll(tasks);
    }
}

static async Task DownloadFileAsync(Attachment attachment, string path) {
    using (var client = new HttpClient()) {
        using (var fileStream = File.Create(path + attachment.FileName)) {
            var downloadedFileStream = await client.GetStreamAsync(attachment.url);
            await downloadedFileStream.CopyToAsync(fileStream);
        }
    }
}

预期希望它一次下载 100 个文件，然后再下载下 100 个；

实际它同时下载了更多。很快就报错了 Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host

Answer 1

就性能而言，以“批处理”方式运行任务并不是一个好主意。长时间运行的任务会使整个批处理阻塞。更好的方法是在任务完成后立即开始一项新任务。

这可以按照@MertAkcakaya 的建议使用队列来实现。但我会根据我的其他答案发布另一个替代方案 Have a set of Tasks with only X running at a time

int maxTread = 3;
System.Net.ServicePointManager.DefaultConnectionLimit = 50; //Set this once to a max value in your app

var urls = new Tuple<string, string>[] {
    Tuple.Create("http://cnn.com","temp/cnn1.htm"),
    Tuple.Create("http://cnn.com","temp/cnn2.htm"),
    Tuple.Create("http://bbc.com","temp/bbc1.htm"),
    Tuple.Create("http://bbc.com","temp/bbc2.htm"),
    Tuple.Create("http://stackoverflow.com","temp/stackoverflow.htm"),
    Tuple.Create("http://google.com","temp/google1.htm"),
    Tuple.Create("http://google.com","temp/google2.htm"),
};
DownloadParallel(urls, maxTread);

---

async Task DownloadParallel(IEnumerable<Tuple<string,string>> urls, int maxThreads)
{
    SemaphoreSlim maxThread = new SemaphoreSlim(maxThreads);
    var client = new HttpClient();

    foreach(var url in urls)
    {
        await maxThread.WaitAsync();
        DownloadFile(client, url.Item1, url.Item2)
                    .ContinueWith((task) => maxThread.Release() );
    }
}


async Task DownloadFile(HttpClient client, string url, string fileName)
{
    var stream = await client.GetStreamAsync(url);
    using (var fileStream = File.Create(fileName))
    {
        await stream.CopyToAsync(fileStream);
    }
}

PS:DownloadParallel 将在开始最后一次下载后立即返回。所以不要等待它。如果你真的想等待它，你应该添加 for (int i = 0; i < maxThreads; i++) await maxThread.WaitAsync();在方法的最后。

PS2:不要忘记将异常处理添加到DownloadFile