json - AWS Data Pipeline - 在创建 EMR 期间设置 Hive 站点值

Question

我们正在将我们的数据管道版本从 3.3.2 升级到 5.8，因此旧 AMI 版本上的那些引导操作已更改为使用配置 设置并在分类/属性定义下指定它们。

所以我的 Json 如下所示

  {
            "enableDebugging": "true",
            "taskInstanceBidPrice": "1",
            "terminateAfter": "2 Hours",
            "name": "ExportCluster",
            "taskInstanceType": "m1.xlarge",
            "schedule": {
                "ref": "Default"
            },
            "emrLogUri": "s3://emr-script-logs/",
            "coreInstanceType": "m1.xlarge",
            "coreInstanceCount": "1",
            "taskInstanceCount": "4",
            "masterInstanceType": "m3.xlarge",
            "keyPair": "XXXX",
            "applications": ["hadoop","hive", "tez"],
            "subnetId": "XXXXX",
            "logUri": "s3://pipelinedata/XXX",
            "releaseLabel": "emr-5.8.0",
            "type": "EmrCluster",
            "id": "EmrClusterWithNewEMRVersion",
            "configuration": [
                { "ref": "configureEmrHiveSite" }
            ]
        },
        {
            "myComment": "This object configures hive-site xml.",
            "name": "HiveSite Configuration",
            "type": "HiveSiteConfiguration",
            "id": "configureEmrHiveSite",
            "classification": "hive-site",
            "property": [
                {"ref": "hive-exec-compress-output" }
            ]
        },
        {
            "myComment": "This object sets a hive-site configuration 
             property value.",
            "name":"hive-exec-compress-output",
            "type": "Property",
            "id": "hive-exec-compress-output",
            "key": "hive.exec.compress.output",
            "value": "true"
        }
    ],
    "parameters": []

使用上面的 Json 文件，它被加载到 Data Pipeline 中，但抛出一个错误

Object:HiveSite Configuration
ERROR: 'HiveSiteConfiguration'
Object:ExportCluster
ERROR: 'configuration' values must be of type 'null'. Found values of type 'null'

我不确定这到底是什么意思，请告诉我我是否正确指定了这个，我认为我是根据 http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html

Answer 1

只有在 AWS 数据管道正确识别它并相应地设置 Hive-site.xml 时，下面的 block 才应将名称命名为“EMR 配置”。

   {
        "myComment": "This object configures hive-site xml.",
        "name": "EMR Configuration",
        "type": "EmrConfiguration",
        "id": "configureEmrHiveSite",
        "classification": "hive-site",
        "property": [
            {"ref": "hive-exec-compress-output" }
        ]
    },