java - 调配 AutoScaling 策略失败 : EMR instance group doesn't exist

Question

我正在使用 AmazonElasticMapReduceAsyncClientBuilder 为实例组设置自动扩展策略。

当我执行 aws emr describe-cluster --cluster-id j-XXXXXXXXX 时，我的自动缩放策略中遇到以下错误。我确实有集群、实例组和 IAM 自动缩放角色 (EMR_AutoScaling_DefaultRole) 可用

"Status": {   "State": "FAILED",   "StateChangeReason": {
    "Message": "Failed to provision the AutoScaling policy: EMR instance group doesn't exist: Failed to find Cluster j-3SGZHX7D2DM98.",
    "Code": "PROVISION_FAILURE"   }

我的 putautoscale 请求是

{   "clusterId": "j-3SGZHX7D2DM98",   "instanceGroupId": "ig-243GEE06N6JXS",   "autoScalingPolicy": {
    "constraints": {
      "minCapacity": 2,
      "maxCapacity": 10
    },
    "rules": [
      {
        "name": "Default-scale-out",
        "description": "Replicates the default scale-out rule in the console for YARN memory by adding one additional instance",
        "action": {
          "simpleScalingPolicyConfiguration": {
            "adjustmentType": "CHANGE_IN_CAPACITY",
            "scalingAdjustment": 1,
            "coolDown": 300
          }
        },
        "trigger": {
          "cloudWatchAlarmDefinition": {
            "comparisonOperator": "LESS_THAN",
            "evaluationPeriods": 1,
            "metricName": "YARNMemoryAvailablePercentage",
            "namespace": "AWS/ElasticMapReduce",
            "period": 200,
            "statistic": "AVERAGE",
            "threshold": 25.0,
            "unit": "PERCENT",
            "dimensions": [
              {
                "key": "JobFlowId",
                "value": "j-3SGZHX7D2DM98"
              }
            ]
          }
        }
      },
      {
        "name": "Default-scale-In",
        "description": "Replicates the default scale-out rule in the console for YARN memory by terminating  one  instance",
        "action": {
          "simpleScalingPolicyConfiguration": {
            "adjustmentType": "CHANGE_IN_CAPACITY",
            "scalingAdjustment": -2,
            "coolDown": 300
          }
        },
        "trigger": {
          "cloudWatchAlarmDefinition": {
            "comparisonOperator": "GREATER_THAN_OR_EQUAL",
            "evaluationPeriods": 1,
            "metricName": "YARNMemoryAvailablePercentage",
            "namespace": "AWS/ElasticMapReduce",
            "period": 300,
            "statistic": "AVERAGE",
            "threshold": 50.0,
            "unit": "PERCENT",
            "dimensions": [
              {
                "key": "JobFlowId",
                "value": "j-3SGZHX7D2DM98"
              }
            ]
          }
        }
      }
    ]   },   "progressListener": {
       },   "requestClientOptions": {
    "markers": {

    },
    "readLimit": 131073   } } 

future 对象的结果

{   ClusterId: j-3SGZHX7D2DM98,   InstanceGroupId: ig-243GEE06N6JXS,   AutoScalingPolicy: {
        Status: {
          State: PENDING,
          StateChangeReason: {
            Code: USER_REQUEST,
            Message: AutoScalingpolicymodifiedbyuserrequest
          }
        },
        Constraints: {
          MinCapacity: 2,
          MaxCapacity: 10
        },
        Rules: [
          {
            Name: Default-scale-out,
            Description: Replicatesthedefaultscale-outruleintheconsoleforYARNmemorybyaddingoneadditionalinstance,
            Action: {
              SimpleScalingPolicyConfiguration: {
                AdjustmentType: CHANGE_IN_CAPACITY,
                ScalingAdjustment: 1,
                CoolDown: 300
              }
            },
            Trigger: {
              CloudWatchAlarmDefinition: {
                ComparisonOperator: LESS_THAN,
                EvaluationPeriods: 1,
                MetricName: YARNMemoryAvailablePercentage,
                Namespace: AWS/ElasticMapReduce,
                Period: 200,
                Statistic: AVERAGE,
                Threshold: 25.0,
                Unit: PERCENT,
                Dimensions: [
                  {
                    Key: JobFlowId,
                    Value: j-3SGZHX7D2DM98
                  }
                ]
              }
            }
          },
          {
            Name: Default-scale-In,
            Description: Replicatesthedefaultscale-outruleintheconsoleforYARNmemorybyterminatingoneinstance,
            Action: {
              SimpleScalingPolicyConfiguration: {
                AdjustmentType: CHANGE_IN_CAPACITY,
                ScalingAdjustment: -2,
                CoolDown: 300
              }
            },
            Trigger: {
              CloudWatchAlarmDefinition: {
                ComparisonOperator: GREATER_THAN_OR_EQUAL,
                EvaluationPeriods: 1,
                MetricName: YARNMemoryAvailablePercentage,
                Namespace: AWS/ElasticMapReduce,
                Period: 300,
                Statistic: AVERAGE,
                Threshold: 50.0,
                Unit: PERCENT,
                Dimensions: [
                  {
                    Key: JobFlowId,
                    Value: j-3SGZHX7D2DM98
                  }
                ]
              }
            }
          }
        ]   } }

状态从待处理更改为失败java代码

Future<PutAutoScalingPolicyResult> putAutoScalingPolicyResultFuture=emr.putAutoScalingPolicyAsync(scalingPolicyRequest);
PutAutoScalingPolicyResultputAutoScalingPolicyResult=emr.putAutoScalingPolicy(scalingPolicyRequest);

我正在做相当于

aws emr put-auto-scaling-policy --cluster-id j-1EKZ3TYEVF1S2 --instance-group-id ig-3PLUZBA6WLS07 --auto-scaling-policy file://your/path/to/autoscaleconfig.json 

谢谢

Answer 1

对我来说，这是因为集群没有标记为“visibleToAllUsers”；我必须发出一个

aws emr modify-instance-attributes --cluster-id BLAH --visible-to-all-users

先改变。