sql-server - Azure 数据工厂在从 SQL 复制到 ADLS 时抛出 'Length Required"错误

Question

我正在尝试通过 Azure 数据工厂 (ADF) 将数据从本地 SQL 服务器复制到 Azure Data Lake Storage (ADLS)。一切似乎都正常，除了当我运行(调试或触发)管道时，我收到错误:

{ "errorCode": "2200", "message": "Failure happened on 'Sink' side. ErrorCode=UserErrorAdlsFileWriteFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Writing to 'AzureDataLakeStore' failed. Message: The remote server returned an error: (411) Length Required.. Response details: \r\nLength Required\r\n\r\n

Length Required

\r\n

---

HTTP Error 411. The request must be chunked or have a content length.

\r\n\r\n,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Net.WebException,Message=The remote server returned an error: (411) Length Required.,Source=System,'", "failureType": "UserError", "target": "CopyData1" }

真正奇怪的是，以下管道确实有效:

• SQL tbl1 -> SQL tbl2
• ADLS 源.txt -> ADLS 接收器.txt

即读/写访问按预期工作。后一个管道还能够创建/覆盖sink.txt 文件。

但是当我运行管道时

• SQL tbl1 -> sink.txt

我收到需要长度错误。如果sink.txt存在，管道甚至会删除它!

我正在使用 ADFv2、ADLS Gen1，ADF 和 ADLS 驻留在同一订阅/资源组中，使用自托管/Azure 集成运行时(分别用于 SQL/ADLS)。我已经使用像“SELECT 1 Col”这样简单的源语句进行了测试。还在没有数据集架构和架构+映射的情况下进行了测试。

这是一个错误，还是我遗漏了什么？需要哪个“长度”？

<小时/>

编辑 1:最小 JSON 脚本

管道1.json

{
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "CopyData1",
                "type": "Copy",
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "typeProperties": {
                    "source": {
                        "type": "SqlSource",
                        "sqlReaderQuery": "SELECT TOP 1 'x' AS col1 FROM sys.tables"
                    },
                    "sink": {
                        "type": "AzureDataLakeStoreSink"
                    },
                    "enableStaging": false,
                    "dataIntegrationUnits": 0
                },
                "inputs": [
                    {
                        "referenceName": "table1",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "sink1",
                        "type": "DatasetReference"
                    }
                ]
            }
        ]
    }
}

表1.json

{
    "name": "table1",
    "properties": {
        "linkedServiceName": {
            "referenceName": "SqlServer1",
            "type": "LinkedServiceReference"
        },
        "type": "SqlServerTable",
        "typeProperties": {
            "tableName": "sys.tables"
        }
    }
}

sink1.json

{
    "name": "sink1",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureDataLakeStore1",
            "type": "LinkedServiceReference"
        },
        "type": "AzureDataLakeStoreFile",
        "structure": [
            {
                "name": "col1",
                "type": "String"
            }
        ],
        "typeProperties": {
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "",
                "nullValue": "\\N",
                "treatEmptyAsNull": true,
                "skipLineCount": 0,
                "firstRowAsHeader": true
            },
            "fileName": "sink1.txt",
            "folderPath": "myDir"
        }
    }
}

<小时/>

编辑2:进行的测试摘要

• SQL -> ADLS 错误
• Oracle -> ADLS 错误
• SQL -> Blob 确定
• Oracle -> Blob 正常
• SQL -> SQL 正常
• ADLS -> ADLS 正常
• AzureSQLDB -> ADLS 正常

Answer 1

您的自托管 IR 是否有一些代理设置或经过特殊的网络设置？当 ADF 的 ADLS 连接器尝试与 ADLS 服务通信时，此类错误应该是由中间代理服务引起的。