amazon-web-services - 如何在多个子网上运行AWS EMR集群？

Question

目前，我们正在使用 EMR 中的 config.json 文件创建实例来配置集群。此文件指定子网(“Ec2SubnetId”)。

所有我的 EMR 实例最终都使用此子网...我如何让它使用多个子网？这是我推送到 S3 的 terraform 模板。

{
   "Applications": [
        {"Name": "Spark"},
        {"Name": "Hadoop"}
    ],
    "BootstrapActions": [
        {
            "Name": "Step1-stuff",
            "ScriptBootstrapAction": {
                "Path": "s3://${artifact_s3_bucket_name}/artifacts/${build_commit_id}/install-stuff.sh",
                "Args": ["${stuff_args}"]
            }
        },
        {
            "Name": "setup-cloudWatch-agent",
            "ScriptBootstrapAction": {
                "Path": "s3://${artifact_s3_bucket_name}/artifacts/${build_commit_id}/setup-cwagent-emr.sh",
                "Args": ["${build_commit_id}"]
            }
        }
    ],
    "Configurations": [
        {
            "Classification": "spark",
            "Properties": {
                "maximizeResourceAllocation": "true"
            }
    ],
    "Instances": {
        "AdditionalMasterSecurityGroups": [ "${additional_master_security_group}" ],
        "AdditionalSlaveSecurityGroups": [ "${additional_slave_security_group}" ],
        "Ec2KeyName": "privatekey-${env}",
        "Ec2SubnetId": "${data_subnet}",
        "InstanceGroups": [

Answer 1

您目前无法实现您想要做的事情。 EMR 集群最终的所有节点始终位于同一子网中。

使用实例队列，您确实能够配置一组子网。但是在启动时，AWS 将选择最好的一个并将您的所有实例放在那里.

来自EMR Documentation ，在“使用控制台配置实例队列”下:

5. For Network, enter a value. If you choose a VPC for Network, choose a single EC2 Subnet or CTRL + click to choose multiple EC2 subnets. The subnets you select must be the same type (public or private). If you choose only one, your cluster launches in that subnet. If you choose a group, the subnet with the best fit is selected from the group when the cluster launches.