Azure 数据工厂管道 + ML

Question

我正在尝试在 Azure 数据工厂 V1 中创建一个管道，它将对文件执行 Azure 批量执行。我使用 blob 存储作为输入和输出来实现它，并且它有效。但是，我并不想将输入和输出更改为数据湖存储中的文件夹。当我尝试部署它时，出现以下错误:

Entity provisioning failed: AzureML Activity 'MLActivity' specifies 'DatalakeInput' in a property that requires an Azure Blob Dataset reference.  

如何将输入和输出作为数据湖存储而不是 blob？

管道:

{
        "name": "MLPipeline",
        "properties": {
            "description": "use AzureML model",
            "activities": [
                {
                    "type": "AzureMLBatchExecution",
                    "typeProperties": {
                        "webServiceInput": "DatalakeInput",
                        "webServiceOutputs": {
                            "output1": "DatalakeOutput"
                        },
                        "webServiceInputs": {},
                        "globalParameters": {}
                    },
                    "inputs": [
                        {
                            "name": "DatalakeInput"
                        }
                    ],
                    "outputs": [
                        {
                            "name": "DatalakeOutput"
                        }
                    ],
                    "policy": {
                        "timeout": "02:00:00",
                        "concurrency": 3,
                        "executionPriorityOrder": "NewestFirst",
                        "retry": 1
                    },
                    "scheduler": {
                        "frequency": "Hour",
                        "interval": 1
                    },
                    "name": "MLActivity",
                    "description": "description",
                    "linkedServiceName": "MyAzureMLLinkedService"
                }
            ],
            "start": "2016-02-08T00:00:00Z",
            "end": "2016-02-08T00:00:00Z",
            "isPaused": false,
            "hubName": "hubname",
            "pipelineMode": "Scheduled"
        }
    }

输出数据集:

  {
        "name": "DatalakeOutput",
        "properties": {
            "published": false,
            "type": "AzureDataLakeStore",
            "linkedServiceName": "AzureDataLakeStoreLinkedService",
            "typeProperties": {
                "folderPath": "/DATA_MANAGEMENT/"
            },
            "availability": {
                "frequency": "Hour",
                "interval": 1
            }
        }
    }

输入数据集:

 {
        "name": "DatalakeInput",
        "properties": {
            "published": false,
            "type": "AzureDataLakeStore",
            "linkedServiceName": "AzureDataLakeStoreLinkedService",
            "typeProperties": {
                "fileName": "data.csv",
                "folderPath": "/RAW/",
                "format": {
                    "type": "TextFormat",
                    "columnDelimiter": ","
                }
            },
            "availability": {
                "frequency": "Hour",
                "interval": 1
            }
        }
    }

AzureDatalakeStoreLinkedService:

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "description": "",
        "hubName": "xyzdatafactoryv1_hub",
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://xyzdatastore.azuredatalakestore.net/webhdfs/v1",
            "authorization": "**********",
            "sessionId": "**********",
            "subscriptionId": "*****",
            "resourceGroupName": "xyzresourcegroup"
        }
    }
}

链接服务是按照此 tutorial 完成的基于数据工厂V1。

Answer 1

我认为 AzureDataLakeStoreLinkedService 存在一些问题。请核实。

根据用于访问数据存储的身份验证，您的 AzureDataLakeStoreLinkedService json 必须如下所示 -

使用服务主体身份验证

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
            "servicePrincipalId": "<service principal id>",
            "servicePrincipalKey": {
                "type": "SecureString",
                "value": "<service principal key>"
            },
            "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
            "subscriptionId": "<subscription of ADLS>",
            "resourceGroupName": "<resource group of ADLS>"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

使用托管服务身份验证

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
            "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
            "subscriptionId": "<subscription of ADLS>",
            "resourceGroupName": "<resource group of ADLS>"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

这是微软引用文档 - Copy data to or from Azure Data Lake Store by using Azure Data Factory