c# - 我收到 WebException : "The operation has timed out" immediately on HttpWebRequest. GetResponse()

Question

我正在多线程环境中大量抓取网页内容。我需要一个可靠的下载器组件，它可以容忍临时服务器故障、连接中断等。下面是我的代码。

现在，我一遍又一遍地遇到一个奇怪的情况:一切都完美地开始了。 10个线程并发拉取数据10分钟左右。在那之后，我在调用我的请求对象的 GetResponse 方法 后立即开始获取 WebException 超时。休息一下(让线程休眠)没有帮助。它仅在我停止应用程序并重新启动它直到接下来的 10 分钟过去并且问题再次出现时才有帮助。

我已经尝试过但没有任何帮助:

• 通过“using”语句显式关闭/处置响应对象
• 调用 request.Abort 可能有帮助的地方
• 在 ServicePointManager/ServicePoint 和 WebRequest 级别处理超时(延长/缩短超时间隔)
• 操纵 KeepAlive 属性
• 调用CloseConnectionGroup
• 操纵同时运行的线程数

没有任何帮助!所以它看起来像是一个错误或至少是非常缺乏记录的行为。我在 Google 和 Stackoverflow 上看到了很多关于此的问题，但没有一个得到完全回答。基本上人们会从上面的列表中推荐其中一个。我都试过了。

    public TResource DownloadResource(Uri uri)
    {
        for (var resourceReadingAttempt = 0; resourceReadingAttempt <= MaxTries; resourceReadingAttempt++)
        {
            var request = (HttpWebRequest)WebRequest.Create(uri);
            HttpWebResponse response = null;
            for (var downloadAttempt = 0; downloadAttempt <= MaxTries; downloadAttempt++)
            {
                if (downloadAttempt > 0)
                {
                    var sleepFor = TimeSpan.FromSeconds(4 << downloadAttempt) + TimeSpan.FromMilliseconds(new Random(DateTime.Now.Millisecond).Next(1000));
                    Trace.WriteLine("Retry #" + downloadAttempt + " in " + sleepFor + ".");
                    Thread.Sleep(sleepFor);
                }
                Trace.WriteLine("Trying to get a resource by URL: " + uri);

                var watch = Stopwatch.StartNew();
                try
                {
                    response = (HttpWebResponse)request.GetResponse();
                    break;
                }
                catch (WebException exception)
                {
                    request.Abort();
                    Trace.WriteLine("Failed to get a resource by the URL: " + uri + " after " + watch.Elapsed + ". " + exception.Message);
                    if (exception.Status == WebExceptionStatus.Timeout)
                    {
                        //Trace.WriteLine("Closing " + request.ServicePoint.CurrentConnections + " current connections.");
                        //request.ServicePoint.CloseConnectionGroup(request.ConnectionGroupName);
                        //request.Abort();
                        continue;
                    }
                    else
                    {
                        using (var failure = exception.Response as HttpWebResponse)
                        {

                            Int32 code;
                            try { code = failure != null ? (Int32)failure.StatusCode : 500; }
                            catch { code = 500; }

                            if (code >= 500 && code < 600)
                            {
                                if (failure != null) failure.Close();
                                continue;
                            }
                            else
                            {
                                Trace.TraceError(exception.ToString());
                                throw;
                            }
                        }
                    }
                }
            }

            if (response == null) throw new ApplicationException("Unable to get a resource from URL \"" + uri + "\".");
            try
            {
                // response disposal is required to eliminate problems with timeouts
                // more about the problem: http://stackoverflow.com/questions/5827030/httpwebrequest-times-out-on-second-call
                // http://social.msdn.microsoft.com/Forums/en/netfxnetcom/thread/a2014f3d-122b-4cd6-a886-d619d7e3140e

                TResource resource;
                using (var stream = response.GetResponseStream())
                {
                    try
                    {
                        resource = this.reader.ReadFromStream(stream);
                    }
                    catch (IOException exception)
                    {
                        Trace.TraceError("Unable to read the resource stream: " + exception.ToString());
                        continue;
                    }
                }
                return resource;
            }
            finally
            {
                // recycle as much as you can
                if (response != null)
                {
                    response.Close();
                    (response as IDisposable).Dispose();
                    response = null;
                }
                if (request != null)
                {
                    //Trace.WriteLine("closing connection group: " + request.ConnectionGroupName);
                    //request.ServicePoint.CloseConnectionGroup(request.ConnectionGroupName);
                    request.Abort();
                    request = null;
                }
            }
        }
        throw new ApplicationException("Resource was not able to be acquired after several attempts.");
    }

Answer 1

我有同样的问题我在网上搜索了很多，我得到了 1 个解决方案，一次固定线程数。你必须控制一次线​​程数，我已经开始使用 2-3 个线程一次。也使用这个 ServicePointManager.DefaultConnectionLimit = 200;这真的对你有帮助。