networking - 步骤 'Publish Artifact: drop' 中的 VSTS 构建代理问题

Question

我们在 vsts 构建中看到不一致的失败。这些都是凭空出现的。

代理在本地运行。使用托管构建代理时，我们看不到任何问题。

在连续构建中，需要多次重试。通常构建会失败，但有时不会。

不同项目中的多个构建定义受到影响

我们正在努力寻找根本原因，任何诊断此问题的帮助将不胜感激!

以下是相关版本信息:

[2017-04-25 07:25:58Z INFO AgentProcess] Agent is built for Windows - win7-x64. [2017-04-25 07:25:58Z INFO AgentProcess] RuntimeInformation: Microsoft Windows 6.3.9600 . [2017-04-25 07:25:58Z INFO AgentProcess] Version: 2.111.1

以下是日志的摘录:

017-04-25T07:28:14.1561247Z ##[section]Starting: Publish Artifact: drop 2017-04-25T07:28:14.1561247Z ============================================================================== 2017-04-25T07:28:14.1561247Z Task : Publish Build Artifacts 2017-04-25T07:28:14.1561247Z Description : Publish Build artifacts to the server or a file share 2017-04-25T07:28:14.1561247Z Version : 1.0.40 2017-04-25T07:28:14.1561247Z Author : Microsoft Corporation 2017-04-25T07:28:14.1561247Z Help : More Information 2017-04-25T07:28:14.1561247Z ============================================================================== 2017-04-25T07:28:15.0779877Z ##[section]Async Command Start: Upload Artifact 2017-04-25T07:28:15.0779877Z Uploading 8 files 2017-04-25T07:28:20.1111153Z Total file: 8 ---- Processed file: 7 (87%) 2017-04-25T07:28:30.2611071Z Total file: 8 ---- Processed file: 7 (87%) 2017-04-25T07:28:35.3611234Z Fail to upload 'E:\Build\agents\scully_work\4\a\Project.Name.Web.zip' due to 'An error occurred while sending the request.'. 2017-04-25T07:28:35.3611234Z System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.WinHttpHandler.d__101.MoveNext() --- End of inner exception stack trace --- at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.d__3.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Net.Http.HttpClient.d__58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult()
at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.<SendAsync>d__45.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.d__17.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.<UploadFileAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.Services.Agent.Worker.Build.FileContainerServer.<UploadAsync>d__14.MoveNext()
2017-04-25T07:28:35.3611234Z Detail upload trace for file that fail to upload: drop/Project.Name.Web.zip
2017-04-25T07:28:35.3611234Z Begin chunking upload file 'drop/Project.Name.Web.zip', chunk size '4194304 Bytes', total chunks '3'.
2017-04-25T07:28:35.3611234Z Attempt '1' for uploading chunk '1' of file 'drop/Project.Name.Web.zip'.
2017-04-25T07:28:35.3611234Z Generate new HttpRequest for uploading file 'drop/Project.Name.Web.zip', chunk '1' of '3'.
2017-04-25T07:28:35.3611234Z Start uploading file 'drop/Project.Name.Web.zip' to server, chunk '1'.
2017-04-25T07:28:35.3611234Z Chunk '1' attempt '1' of file 'drop/Project.Name.Web.zip' fail to send request to server. Error: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: The connection with the server was terminated abnormally
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.WinHttpHandler.<StartRequest>d__101.MoveNext()
--- End of inner exception stack trace ---
at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.<SendAsync>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Net.Http.HttpClient.<FinishSendAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.d__45.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at Microsoft.VisualStudio.Services.FileContainer.Client.FileContainerHttpClient.d__17.MoveNext(). 2017-04-25T07:28:35.3611234Z Backoff 5.16 seconds before attempt '2' chunk '1'

更新:
将代理更新到最新的稳定版，问题仍然存在:

[2017-04-25 08:33:04Z INFO AgentProcess] Agent is built for Windows - win7-x64. [2017-04-25 08:33:04Z INFO AgentProcess] RuntimeInformation: Microsoft Windows 6.3.9600 . [2017-04-25 08:33:04Z INFO AgentProcess] Version: 2.115.0

Answer 1

这通常意味着在您的代理和您遇到中断的服务之间存在网络问题。

从 2.x 代理开始，我们积极尝试处理网络状况。我们将每个文件分成 block (4MB)，打开一个 http 流并将该文件 block 通过管道传输到服务。每个文件 block 都有很长的时间(10分钟)

我们有每 block 重试和退避重试。

Backoff 5.16 seconds before attempt '2' chunk '1'

然后，如果即使重试仍然无法上传 block ，我们将该文件放入重试队列以再通过一次。

如您所见，我们非常努力。

网络中断越多，文件越大，几率就越大。此外，如果上传速度非常慢，则可能无法及时上传 block (10 分钟内 4mb 是很长的时间)，但在您的情况下，看起来流/网络正在中断。

你应该检查网络。您的机器和服务之间出现了中断。

另一个要尝试的实验是私有(private)云 VM(如 azure)上的私有(private)代理。这为您提供了更快的上传云网络，还可以控制私有(private)代理(安装您想要的软件、增量源和打包等......)。

你看到我描述的重试了吗？我也将此转发给我们的代理开发人员。如果还有什么要补充的，我会编辑。