yaml - 如何在 CloudWatch 警报中传递 EMR 集群 ID

Question

我正在尝试为我的 EMR 集群创建 SNS 警报，因此当 EMR 集群发生故障时我应该收到通知。

但我的问题是我无法在 CloudWatch Alarm 中将集群 ID 作为 JobFlowId 传递。

我使用 CloudFomartion Templatet 创建所有资源。

当我使用 REF 引用 Cluster Id 时，出现以下错误。

Error
Template validation error: Template format error: Unresolved resource dependencies [FinancialLineItemEmrCluster] in the Resources block of the template

这是我的模板。

我在创建资源 EMRAlarm 时特别在 JobFlowId 处遇到问题

AWSTemplateFormatVersion: '2010-09-09'
Description: TRF SDI Optmization Full File  Creation

Parameters:
  AppName:
    Default: trfsdioptmization
    Description: trfsdioptmization.
    Type: String
  Environment:
    Type: String
    Default: nonprod
  FinancialIdentifier:
    Type: String
    Default: 123456789
  ApplicationAssetInsightId:
    Type: String
    Default: 12345678
  EnvironmentType:
    Type: String
    AllowedValues:
      - "prod"
      - "PRE-PRODUCTION"
      - "QUALITY ASSURANCE"
      - "nonprod"
    Default: nonprod
  ResourceOwner:
    Type: String
    Default: <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="87f4f2e3e6f5f4efe6e9a9ecf2eae6f5c7e6e5e4e3a9e4e8ea" rel="noreferrer noopener nofollow">[email protected]</a>
  EnvironmentPhase:
    Type: String
    Default: nonprod
  RegionAbbreviation:
    Default: us-east-1
    Description: Region Abbreviation e.g. us-east-1 for us-east
    Type: String

Resources:
 TRFSDIFullfileGeneration: 
  Type: "AWS::DataPipeline::Pipeline"
  #DeletionPolicy: Retain
  Properties: 
    Name: !Sub "${ApplicationAssetInsightId}-tr-fr-${EnvironmentPhase}-${RegionAbbreviation}-${AppName}-DataPipeline"
    Description: "Pipeline to create full file for TRFSDI full file Optmization"
    Activate: false
    PipelineObjects: 
      - 
        Id: "FinancialLineItemActivity"
        Name: "FinancialLineItemActivity"
        Fields: 
          - 
            Key: "type"
            StringValue: "EmrActivity"
          - 
            Key: "runsOn"
            RefValue: "FinancialLineItemEmrCluster"
          - 
            Key: "step"
            StringValue: "command-runner.jar,spark-submit,--master,yarn-cluster,--deploy-mode,cluster,--class,start.EntryFileCreation,s3://205147-trf-fr-nonprdo-us-east-1-trfsdioptmization/AJAR/SparkJob-0.1-jar-with-dependencies.jar,FinancialLineItem"

        Id: "Default"
        Name: "Default"
        Fields: 
          - 
            Key: "type"
            StringValue: "Default"
          - 
            Key: "scheduleType"
            StringValue: "ONDEMAND"
          - 
            Key: "failureAndRerunMode"
            StringValue: "CASCADE"
          - 
            Key: "role"
            StringValue: "DataPipelineDefaultRole"
          - 
            Key: "resourceRole"
            StringValue: "DataPipelineDefaultResourceRole"
          - 
            Key: "pipelineLogUri"
            StringValue: "s3://205147-tr-fr-nonprod-us-east-1-trfsdioptmization/EMRLOGS"
      - 
        Id: "FinancialLineItemEmrCluster"
        Name: "FinancialLineItemEmrCluster"
        Fields: 
          - 
            Key: "terminateAfter"
            StringValue: "30 Minutes"
          - 
            Key: "releaseLabel"
            StringValue: "emr-5.9.0"
          - 
            Key: "masterInstanceType"
            StringValue: "m3.xlarge"
          - 
            Key: "coreInstanceType"
            StringValue: "m3.2xlarge"
          - 
            Key: "coreInstanceCount"
            StringValue: "2"
          - 
            Key: "type"
            StringValue: "EmrCluster"
          - 
            Key: "applications"
            StringValue: "spark"
          - 
            Key: "subnetId"
            StringValue: "subnet-86febcab"
          - 
            Key: "onSuccess"
            RefValue: "FinancialLineItem_Success"
          - 
            Key: "onFail"
            RefValue: "FinancialLineItem_Fail"
 EMRAlarm:
   Type: AWS::CloudWatch::Alarm
   Properties:
    AlarmDescription: "Raise alarm if apps running on EMR cluster is killed"
    Namespace: AWS/ElasticMapReduce
    MetricName: AppsKilled
    Dimensions:
    - Name: 205147-TRFSDIOPTmization
      JobFlowId: !Ref FinancialLineItemEmrCluster
    Statistic: Average
    Period: 15
    EvaluationPeriods: '1'
    ComparisonOperator: GreaterThanOrEqualToThreshold
    Threshold: 1
    AlarmActions:
    - "AWSTRF_HEALTH"

Answer 1

您的模板中没有 AWS::EMR::Cluster 类型的资源。您在 CloudWatch 警报中引用了名为 FinancialLineItemEmrCluster 的内容。从上下文来看，我假设您正在尝试引用 EMR 工作。但是，由于您的模板中没有名为 FinancialLineItemEmrCluster 的参数或资源，因此您无法访问 is。

EMR 集群 ID 通常如下所示:j-1ABCD123AB1A。您有多种选择:

如果此集群位于另一个模板中，您可以创建 CloudFormation export在该堆栈中，并使用 !ImportValue在您的模板中导入它。

或者，您可以在模板中使用参数并以这种方式传递 ClusterId。将其添加为参数:

示例:

 FinancialLineItemEmrCluster:
    Description: 'Your EMR cluster id. Example: j-1ABCD123AB1A'
    Type: String

第三种选择是将其硬编码到您的模板中。

无论如何，您都不能直接引用另一个堆栈中的资源。

如果您尚未创建 EMR::Cluster，并且没有 Cluster Id，则需要先创建一个。您可以使用 AWS::EMR::Cluster 将其添加到您的模板中资源:

FinancialLineItemEmrCluster: 
  Type: AWS::EMR::Cluster
  Properties: 
    Instances: 
      MasterInstanceGroup: 
        InstanceCount: 1
        InstanceType: "m3.xlarge"
        Market: "ON_DEMAND"
        Name: "Master"
      CoreInstanceGroup: 
        InstanceCount: 2
        InstanceType: "m3.xlarge"
        Market: "ON_DEMAND"
        Name: "Core"
      TerminationProtected: true
    Name: "TestCluster"
    JobFlowRole: "EMR_EC2_DefaultRole"
    ServiceRole: "EMR_DefaultRole"
    ReleaseLabel: "emr-4.2.0"