azure - Vertex 在 ADLA 中的准备时间过长

Question

我有一个 U-SQL 脚本，它在数据湖中存储的多个 JSON 文件上运行，当我针对单个文件运行该脚本时，一切正常，但是打开它以针对我的所有文件运行每次都会出现以下错误，并且作业永远不会超过“准备”阶段(该阶段花费大约 30 分钟):

System: E_SYSTEM_COMPILER_COMPILERINTERNALERROR
SEVERITY: ERROR
DESCRIPTION: Compilation failed due to internal system error.
MESSAGE: Compilation failed due to internal system error. (255)
DETAILS: Compilation failed for Deep.; JobId: 6ddc1049-c064-46d4-    b31f-1073ffd2cc99; Duration: 00:31:10.4668250; Exit Code: 255; Configuration: kobo_vnext_signed_5395848; Runtime: kobo_vnext_signed_5395848; Output: Deleted file D:\data\CCS\jobs\6ddc1049-c064-46d4-b31f-1073ffd2cc99_v0\securityInfo.
Deleted file D:\data\CCS\jobs\6ddc1049-c064-46d4-b31f-1073ffd2cc99_v0\jobEncryptionKey.
Server Version: 10.2.2.0
Client Version: 10.2.2.0
Target: Cluster
Stat: FetchReal
SkipPath: False
Syntax Parsing finished.
Resource Resolution finished.
Input/Output Resolution finished.
HandleCodeBehind finished.
Wait until structured stream metadata loading end finished.
Binder Pass finished.
Semantic Analyzer finished.
Error Reporter finished.
OptTree Generator finished.

--==== CQO Optimization ====--
Traceflag 608 is ON
Traceflag 551 is ON
Traceflag 402 is ON
Traceflag 550 is ON
Traceflag 21 is ON
Traceflag 552 is ON
Traceflag 554 is ON
Traceflag 515 is ON
Traceflag 534 is ON
Traceflag 541 is ON
Traceflag 435 is ON
Traceflag 106 is ON
Traceflag 62 is ON
Traceflag 571 is ON
Traceflag 514 is ON
Traceflag 568 is ON
Traceflag 569 is ON
Traceflag 203 is ON
Traceflag 546 is ON
Traceflag 562 is ON
Traceflag 563 is ON
Traceflag 505 is ON
Traceflag 506 is ON
Traceflag 542 is ON
Polaris scale up query execution is not enabled
Timer[CQO - Init] = 0.0172413 s
Timer[CQO - CopyIn] = 0.154232 s
Timer[CQO - Search - Init] = 0.000229223 s
Timer[CQO - Search - NNF] = 0.0124772 s
Timer[CQO - Search - PreOptimization] = 0.000576905 s
Pruning partitions and streamsets
Simplifying relational tree

Timer[CQO - Total] = 749.761 s
CQO: Optimization failed.


;

RESOLUTION

在 VS2017 的“诊断”选项卡上我还看到:

Vertex ran too long: 00:31:10

什么可能导致这种情况？我正在努力解决这个问题

Answer 1

您可能想尝试 FileSet 选项 ( https://github.com/Azure/AzureDataLake/blob/master/docs/Release_Notes/2017/2017_04_24/USQL_Release_Notes_2017_04_24.md#input-file-set-scales-orders-of-magnitudes-better-opt-in-statement-is-now-provided )

几周前我遇到了类似的问题。我首先通过增加AU的数量来解决这个问题。但是然后，使用

 SET @@FeaturePreviews = "FileSetV2Dot5:on";

有助于减少编译时间，同时使用较少数量的 AU