Azure 数据工厂副本的 MySQL 超时

Question

我正在使用 azure 数据工厂从 MySQL 服务器复制数据作为源。数据量很大。当我设置管道并执行它时:

MySQL: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

我认为这可以通过 this answer 来解决。如何使用 MySQL 作为源将此配置添加到我的数据工厂管道？

更新:我使用普通脚本将数据从本地 MySQL 复制到 SQL 数据仓库。 MySQL 查询很简单:select * from mytable;完整错误:

Copy activity encountered a user error at Source side: GatewayNodeName=MYGATEWAY,ErrorCode=UserErrorFailedMashupOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message='Type=Microsoft.Data.Mashup.MashupValueException,Message=MySQL: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.,Source=Microsoft.MashupEngine,',Source=,'.

Answer 1

好吧，如果此问题与默认超时配置有关，您可以在管道设置中的“事件” 中添加这些脚本，将超时设置为 1 小时:

"Policy": {
       "concurrency": 1,
       "timeout": "01:00:00"
 }

----------更新----------

管道配置的整个 JSON 如下:

{
     "name": "ADFTutorialPipelineOnPrem",
     "properties": {
     "description": "This pipeline has one Copy activity that copies data from an on-prem SQL to Azure blob",
     "activities": [
       {
         "name": "CopyFromSQLtoBlob",
         "description": "Copy data from on-prem SQL server to blob",
         "type": "Copy",
         "inputs": [
           {
             "name": "EmpOnPremSQLTable"
           }
         ],
         "outputs": [
           {
             "name": "OutputBlobTable"
           }
         ],
         "typeProperties": {
           "source": {
             "type": "SqlSource",
             "sqlReaderQuery": "select * from emp"
           },
           "sink": {
             "type": "BlobSink"
           }
         },
         "Policy": {
           "concurrency": 1,
           "executionPriorityOrder": "NewestFirst",
           "style": "StartOfInterval",
           "retry": 0,
           "timeout": "01:00:00"
         }
       }
     ],
     "start": "2016-07-05T00:00:00Z",
     "end": "2016-07-06T00:00:00Z",
     "isPaused": false
   }
 }

以下示例假设您已在 MySQL 中创建了一个表“MyTable”，并且它包含一个名为“timestampcolumn”的列，用于存储时间序列数据。设置“external”:“true”通知数据工厂服务该表位于外部数据工厂并且不是由数据工厂中的事件生成的。:

{
        "name": "MySqlDataSet",
        "properties": {
            "published": false,
            "type": "RelationalTable",
            "linkedServiceName": "OnPremMySqlLinkedService",
            "typeProperties": {},
            "availability": {
                "frequency": "Hour",
                "interval": 1
            },
            "external": true,
            "policy": {
                "externalData": {
                    "retryInterval": "00:01:00",
                    "retryTimeout": "01:00:00",
                    "maximumRetry": 3
                }
            }
        }
    }

有关如何为Azure数据工厂创建管道的更多详细信息，请参阅this document

有关将数据从本地 MySQL 移动到 Azure 数据工厂的整个教程的更多信息，请参阅 this link .