Azure ML 作业开始失败，出现 DataAccessError : Invalid secret token header

Question

我在圣诞节前一直在运行完全相同的 Azure ML 作业，但昨天它在尝试从数据存储装载 uri_folder 时开始抛出 DataAccessError。我没有更改任何设置，存储帐户 key 是相同的，并且实际上没有为 Azure ML 中的数据存储设置身份验证。我已经从日志中隐藏了相关信息，但没有引入上个月的工作和昨天的工作之间的变化。

我正在使用 Azure CLI 2.39.0 使用相同的 YAML 模板启 Action 业。

我注意到作业之间的唯一变化是 azureml_dataprep 版本，之前是 4.8.1，现在是 4.8.3。让我分享 AzureML 作业日志中的 data_capability.log 片段:

INFO 2023-01-05 11:12:28,625 cr_utilities.py:434 [30] - 安装的 python 软件包版本。 {'azureml_core': '1.45.0.post3', 'azureml_dataprep': '4.8.3', 'azureml_dataprep_rslex': '2.15.1',​​...}

日志的其余部分只是用粗体字符注释了错误描述:

“异常”:“DataAccessError(未知(“解析 AccessToken ClientAuthenticationError 时发生意外错误:意外的内容类型\“text/plain; charset=utf-8\”\n内容:无效的 secret token header :XXXX “，无))”，

**ERROR 2023-01-04 11:34:14,976 cr_utilities.py:157 [28] - ##[error] Data capability failure**
{
   "exception":"PyFuseError",
   "subscription_id":"XXXX",
   "resource_group":"XXXX",
   "workspace_name":"XXXX",
   "workspace_location":"XXXX",
   "experiment_name":"azureml",
   "node_rank":"None",
   "node_id":"tvmps_b0220a4fc0d982fccb02adbe37176fd32d427a576e356b4f56cc806efd97ab8b_d",
   "root_run_id":"XXXX",
   "run_id":"XXXX",
   "correlation_id":"XXXX",
   "source":"common_runtime.data-capability",
   "session_id":"8e522bb0-c461-442b-8342-c2a0609bf30a",
   "version":"0.0.1.20221213.1",
   "commit":"XXXX",
   "branch":"origin/XXXX",
   "os":"linux",
   "TraceId":"538b20ae3ae639547c41eb120ac98b60",
   "SpanId":"3bed598aa514feb7",
   "TraceFlags":"01",
   "traceId":"538b20ae3ae639547c41eb120ac98b60",
   "spanId":"3bed598aa514feb7",
   "custom_dimensions":{
      "exception":"PyFuseError",
      "subscription_id":"XXXX",
      "resource_group":"XXXX",
      "workspace_name":"XXXX",
      "workspace_location":"XXXX",
      "experiment_name":"azureml",
      "node_rank":"None",
      "node_id":"tvmps_b0220a4fc0d982fccb02adbe37176fd32d427a576e356b4f56cc806efd97ab8b_d",
      "root_run_id":"XXXX",
      "run_id":"XXXX",
      "correlation_id":"XXXX",
      "source":"common_runtime.data-capability",
      "session_id":"8e522bb0-c461-442b-8342-c2a0609bf30a",
      "version":"0.0.1.20221213.1",
      "commit":"XXXX",
      "branch":"origin/XXXX",
      "os":"linux",
      "TraceId":"538b20ae3ae639547c41eb120ac98b60",
      "SpanId":"3bed598aa514feb7",
      "TraceFlags":"01",
      "traceId":"538b20ae3ae639547c41eb120ac98b60",
      "spanId":"3bed598aa514feb7"
   }
}



**ERROR 2023-01-04 11:34:14,977 capability_session.py:102 [28] - ##[error] [CapabilitySession][start] Failed to start data session.** 

{
   "session_name":"XXXX",
   **"exception":"DataAccessError(Unknown(\"an unexpected error occurred while resolving AccessToken ClientAuthenticationError: Unexpected content type \\\"text/plain; charset=utf-8\\\"\\nContent: Invalid secret token header: XXXX\", None))",**
   "subscription_id":"XXXX",
   "resource_group":"XXXX",
   "workspace_name":"XXXX",
   "workspace_location":"XXXX",
   "experiment_name":"azureml",
   "node_rank":"None",
   "node_id":"tvmps_b0220a4fc0d982fccb02adbe37176fd32d427a576e356b4f56cc806efd97ab8b_d",
   "root_run_id":"XXXX",
   "run_id":"XXXX",
   "correlation_id":"XXXX",
   "source":"common_runtime.data-capability",
   "session_id":"8e522bb0-c461-442b-8342-c2a0609bf30a",
   "version":"0.0.1.20221213.1",
   "commit":"XXXX",
   "branch":"origin/XXXX",
   "os":"linux",
   "TraceId":"538b20ae3ae639547c41eb120ac98b60",
   "SpanId":"3bed598aa514feb7",
   "TraceFlags":"01",
   "traceId":"538b20ae3ae639547c41eb120ac98b60",
   "spanId":"3bed598aa514feb7",
   "custom_dimensions":{
      "session_name":"XXXX",
      **"exception":"DataAccessError(Unknown(\"an unexpected error occurred while resolving AccessToken ClientAuthenticationError: Unexpected content type \\\"text/plain; charset=utf-8\\\"\\nContent: Invalid secret token header: XXXX\", None))"**,
      "subscription_id":"XXXX",
      "resource_group":"XXXX",
      "workspace_name":"XXXX",
      "workspace_location":"XXXX",
      "experiment_name":"azureml",
      "node_rank":"None",
      "node_id":"tvmps_b0220a4fc0d982fccb02adbe37176fd32d427a576e356b4f56cc806efd97ab8b_d",
      "root_run_id":"XXXX",
      "run_id":"XXXX",
      "correlation_id":"XXXX",
      "source":"common_runtime.data-capability",
      "session_id":"8e522bb0-c461-442b-8342-c2a0609bf30a",
      "version":"0.0.1.20221213.1",
      "commit":"XXXX",
      "branch":"origin/XXXX",
      "os":"linux",
      "TraceId":"538b20ae3ae639547c41eb120ac98b60",
      "SpanId":"3bed598aa514feb7",
      "TraceFlags":"01",
      "traceId":"538b20ae3ae639547c41eb120ac98b60",
      "spanId":"3bed598aa514feb7"
   }
}

这个月有什么变化，或者由于年份变化而需要考虑的事情吗？我还检查了

• 存储帐户 key 尚未重新生成。
• 还执行了 az ml Workspacesync-keys 来刷新 key 以防万一。
• 我能够探索数据存储区，并尝试从 Azure ML 门户的作业装载数据。

提前致谢!!

Answer 1

好的，似乎有什么东西损坏了分配给用户并具有系统分配身份的两个计算实例。使用与之前使用的设置相同的设置创建一个新的计算实例解决了该问题。