amazon-web-services - 使用一个 cloudFormation 模板从 zip 文件创建一个 lambda 并创建一个 S3 存储桶

Question

如何在同一 cloudormation 模板中创建 S3 存储桶和 lambda？

lambda 有很多代码行，因此无法内联编码。通常我将 lambda zip 上传到 S3 存储桶，然后指定 zip 的 S3 key 以在我的 cloudFormation 模板中创建 lambda。如何在无需事先手动创建 S3 存储桶的情况下执行此操作？基本上我要问的是，AWS 中是否有一个临时存储选项可用于上传文件，而无需手动创建 S3 存储桶。

我尝试在线搜索，但所有结果都指向将 zip 文件上传到 S3 存储桶，并在 cloudFormation 模板中使用该文件来创建 lambda。这在这里不起作用，因为 S3 存储桶也是在同一 cloudFormation 模板中创建的。

Answer 1

您可以执行如下操作，创建一个 S3 存储桶、一个 lambda 函数、压缩内联代码并创建一个事件通知，如果对象上传到指定存储桶，该事件通知将触发 lambda 函数。我还添加了一个事件通知，您可以相应地忽略或删除它。

确保在 lambda 函数中将您的代码片段替换为我的代码片段。

据我所知，要么您必须创建 S3 存储桶，预先将文件上传到其中，然后使用这些详细信息在 lambda 函数中指向您的 zip 文件。或者首先通过 lambda 创建 S3 存储桶，然后在配置资源后手动将文件上传到其中。

在我的 lambda 函数中，您可以注意到我已经为 zip 提供了倾斜代码，但如果您已经拥有存储桶，您仍然可以提供 S3 存储桶和 key 。

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-code.html

您还可以查看this他们动态创建了一个 S3 对象并指向创建的存储桶。不过我没有亲自测试过，所以你可以测试一下是否也可以上传zip文件。

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  LambdaFunctionName:
    Type: String
    MinLength: '1'
    MaxLength: '64'
    AllowedPattern: '[a-zA-Z][a-zA-Z0-9_-]*'
    Description: The name of the Lambda function to be deployed
    Default: convert_csv_to_parquet_v2
  LambdaRoleName:
    Type: String
    MinLength: '1'
    MaxLength: '64'
    AllowedPattern: '[\w+=,.@-]+'
    Description: The name of the IAM role used as the Lambda execution role
    Default: Lambda-Role-CFNExample
  LambdaPolicyName:
    Type: String
    MinLength: '1'
    MaxLength: '128'
    AllowedPattern: '[\w+=,.@-]+'
    Default: Lambda-Policy-CFNExample
  NotificationBucket:
    Type: String
    Description: S3 bucket that's used for the Lambda event notification

Resources:
  ExampleS3:
    Type: AWS::S3::Bucket
    DependsOn: LambdaInvokePermission
    Properties:
      BucketName: !Ref NotificationBucket
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: s3:ObjectCreated:Put
            Filter:
              S3Key:
                Rules:
                  - Name: suffix
                    Value: txt
            Function: !GetAtt LambdaFunction.Arn
  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Ref LambdaRoleName
      Description: An execution role for a Lambda function launched by CloudFormation
      ManagedPolicyArns:
        - !Ref LambdaPolicy
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service: lambda.amazonaws.com
          Action:
          - 'sts:AssumeRole'
      
  LambdaPolicy:
    Type: AWS::IAM::ManagedPolicy
    Properties:
      ManagedPolicyName: !Ref LambdaPolicyName
      Description: Managed policy for a Lambda function launched by CloudFormation
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action:
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
            Resource: !Join ['',['arn:', !Ref AWS::Partition, ':logs:', !Ref AWS::Region, ':', !Ref AWS::AccountId, ':log-group:/aws/lambda/', !Ref LambdaFunctionName, ':*']]
          - Effect: Allow
            Action:
              - 'logs:CreateLogGroup'
            Resource: !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:*'
          - Effect: Allow
            Action:
              - 's3:*'
            Resource: '*'
        
  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Join ['',['/aws/lambda/', !Ref LambdaFunctionName]]
      RetentionInDays: 30
            
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Read CSV files from a S3 location and converting them into Parquet
      FunctionName: !Ref LambdaFunctionName
      Handler: lambda_function.lambda_handler
      MemorySize: 128
      Runtime: python3.9
      Role: !GetAtt 'LambdaRole.Arn'
      Timeout: 60
      Code:
        ZipFile: |
            # Imports
            import pandas
            from urllib.parse import unquote_plus
            import boto3
            import os            
            
            def lambda_handler(event, context):
                print(f'event >> {event}')
                s3 = boto3.client('s3', region_name='us-east-1')
            
                for record in event['Records']:
                    key = unquote_plus(record['s3']['object']['key'])
                    print(f'key >> {key}')
            
                    bucket = unquote_plus(record['s3']['bucket']['name'])
                    print(f'bucket >> {bucket}')
            
                    get_file = s3.get_object(Bucket=bucket, Key=key)
                    get = get_file['Body']
                    print(f'get >> {get}')
            
                    df = pandas.DataFrame(get)
                    print('updating columns..')
                    df.columns = df.columns.astype(str)
            
                    print('saving file to s3 location...')
                    df.to_parquet(f's3://csvtoparquetconverted/{key}.parquet')
                    print('file converted to parquet')
  LambdaInvokePermission:
    Type: 'AWS::Lambda::Permission'
    Properties:
      FunctionName: !GetAtt LambdaFunction.Arn
      Action: 'lambda:InvokeFunction'
      Principal: s3.amazonaws.com
      SourceAccount: !Ref 'AWS::AccountId'
      SourceArn: !Sub 'arn:aws:s3:::${NotificationBucket}'

Outputs:
  CLI:
    Description: Use this command to invoke the Lambda function
    Value: !Sub |
        aws lambda invoke --function-name ${LambdaFunction} --payload '{"null": "null"}' lambda-output.txt --cli-binary-format raw-in-base64-out