c# - 服务总线触发的 Azure 功能因 System.Threading.Tasks.TaskCanceledException 失败

Question

我们有一个非常简单的 .NET 隔离 v4 Azure 函数应用，运行 .NET7，它具有三个函数，全部使用 ServiceBusTrigger。

我们发现一小部分执行失败，但存在以下异常:

System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess 处的 System.Threading.Tasks.TaskCanceledException

异常似乎是从函数自己的框架内部抛出的，而不是我们自己的代码。下面是完整的堆栈跟踪。网络搜索表明，当存在未等待的异步任务时，可能会发生这种情况，但我已经确认我们的代码中不存在此类问题。

每个函数执行以下简单任务:

1. 反序列化消息内容
2. 调用和 HTTP 端点
3. 将指标记录到 App Insights

更糟糕的是，这些异常不会导致服务总线消息成为死信，就像我们自己的代码引发异常的情况一样。因此，我们正在丢失关键的产品数据。

有人知道为什么会发生这种情况吗？

这是完整的堆栈跟踪:

{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":0,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":1,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Script, Version=4.15.0.0, Culture=neutral, PublicKeyToken=null","method":"Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker+<InvokeCore>d__9.MoveNext","level":2,"line":96,"fileName":"/src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":3,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":4,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":5,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Script, Version=4.15.0.0, Culture=neutral, PublicKeyToken=null","method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext","level":6,"line":82,"fileName":"/src/azure-functions-host/src/WebJobs.Script/Description/FunctionInvokerBase.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":7,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":8,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":9,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext","level":10,"line":20,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":11,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":12,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":13,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__10.MoveNext","level":14,"line":52,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":15,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":16,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":17,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__33.MoveNext","level":18,"line":581,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":19,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":20,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":21,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__32.MoveNext","level":22,"line":527,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":23,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":24,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":25,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext","level":26,"line":306,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":27,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__26.MoveNext","level":28,"line":352,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw","level":29,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess","level":30,"line":0}
{"assembly":"System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e","method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification","level":31,"line":0}
{"assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.36.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35","method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__18.MoveNext","level":32,"line":108,"fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"}

更新

GitHub Azure Functions 主机问题官方论坛上的相同问题 - 9147

Answer 1

请检查以下发现是否有帮助:

System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
/src/azure-functions-host/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw
C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs
\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs

我已经看到了您给出的完整错误信息(Stack Trace)，其中重复包含 4 到 5 个常见错误。

根据我的研究，所有错误消息都是由于 HttpClient 上的 Timeout 属性而产生的，并确保底层连接不应关闭。

Each function performs the following simple tasks:

1. Deserialize the message content
2. Call and HTTP endpoint
3. Log a metric to App Insights

检查您的 Http 端点请求和响应花费的时间或连接之间的任何超时。

我发现了一些 SO 问题 1 , 2 & 3使用类似的异常消息和给出的解决方案来检查函数触发器执行中的底层 Http 客户端连接。

此外，即使集成了 App Insights 实例，并且在功能代码中实现了记录器类，每个异常消息都会记录在 Application Insights 中，您可以在其中分析完整的遥测数据，有关此 MS Doc 的信息.