BucketDeployment creates multiple identical, unused AwsCliLayers #32907

wimlewis-amazon · 2025-01-14T00:48:04Z

If a stack contains several BucketDeployment constructs, even though the construct will only create one underlying custom resource provider and lambda, it will create a separate AwsCliLayer for each construct. All but one of these lambda layers will be unused.

You can see the bug in the constructor for BucketDeployment in bucket-deployment.ts. It invokes new BucketDeploymentSingletonFunction(...), which I assume is in charge of not creating multiple identical resource provider lambdas. However, one of the values passed to that function is layers: [new AwsCliLayer(this, 'AwsCliLayer')] (link to code). This value is evaluated before the singleton function logic is called, which means that a new AwsCliLayer is created unconditionally. Only the first one will actually be referenced by any lambda! This can be confirmed by examining the synthesized template or by looking at the lambda layers in the AWS console.

In addition to being wasteful, this is also causing deployment failures for us, because CFN attempts to create/update all of those layers simultaneously, resulting in a throttling error ("Resource handler returned message: "Rate exceeded (Service: AWSLambdaInternal; Status Code: 400; Error Code: ThrottlingException ..."). This may be the cause of issue #26940, possibly, although their description sounds a little bit different from ours.

What I think should happen instead: Only one AwsCliLayer needs to exist per stack. This can be referenced by multiple BucketDeplyments' lambda handlers. Note that even though BucketDeployment may create multiple "singletons" for different configurations (EFS, ephemeral-storage size, etc etc), they can all share the same AwsCliLayer.

The text was updated successfully, but these errors were encountered:

ashishdhingra · 2025-01-21T18:19:28Z

Needs reproduction.

ashishdhingra · 2025-01-21T19:05:08Z

Reproducible using below steps:

In the CDK project, under lib folder where TypeScript file defining stack is present, create 3 folders and text files as below:

Folder Name Text file Name Text file content

source-files-1 testsourcefile1.txt Testing

source-files-2 testsourcefile2.txt Testing

source-files-3 testsourcefile3.txt Testing

Define CDK stack (file named lib/cdktest-stack-new.ts:

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as s3deploy from 'aws-cdk-lib/aws-s3-deployment';
import path = require('path');

export class CdktestStackNew extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const destinationBucket = s3.Bucket.fromBucketAttributes(this, 'DestinationBucket', { bucketName: 'some-testbucket' });
    new s3deploy.BucketDeployment(this, 'DeployFiles1', {
      sources: [s3deploy.Source.asset(path.join(__dirname, 'source-files-1'))],
      destinationBucket,
    });

    new s3deploy.BucketDeployment(this, 'DeployFiles2', {
      sources: [s3deploy.Source.asset(path.join(__dirname, 'source-files-2'))],
      destinationBucket,
    });

    new s3deploy.BucketDeployment(this, 'DeployFiles3', {
      sources: [s3deploy.Source.asset(path.join(__dirname, 'source-files-3'))],
      destinationBucket,
    });
  }
}

Define CDK app in bin/cdktest.ts:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdktestStackNew } from '../lib/cdktest-stack-new';

const app = new cdk.App();

new CdktestStackNew(app, 'CdktestStackNew', {
  env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
});

const assembly = app.synth();

Run cdk synth.

Result:
Running cdk synth synthesizes below CloudFormation template:

Resources:
  DeployFiles1AwsCliLayer15AA725E:
    Type: AWS::Lambda::LayerVersion
    Properties:
      Content:
        S3Bucket: cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
        S3Key: 09ee37fb9a746a9dea0618b3024913848346b4b63d08d3fae387b9a11ac83a01.zip
      Description: /opt/awscli/aws
    Metadata:
      aws:cdk:path: CdktestStackNew/DeployFiles1/AwsCliLayer/Resource
      aws:asset:path: asset.09ee37fb9a746a9dea0618b3024913848346b4b63d08d3fae387b9a11ac83a01.zip
      aws:asset:is-bundled: false
      aws:asset:property: Content
  DeployFiles1CustomResource5301B660:
    Type: Custom::CDKBucketDeployment
    Properties:
      ServiceToken:
        Fn::GetAtt:
          - CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C81C01536
          - Arn
      SourceBucketNames:
        - cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
      SourceObjectKeys:
        - bfa1ca4934d7fa9441dc6586957ccfe9d6ec55046f82cdfeba297f71d5549c27.zip
      DestinationBucketName: some-testbucket
      Prune: true
      OutputObjectKeys: true
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: CdktestStackNew/DeployFiles1/CustomResource/Default
  CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRole89A01265:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRole
            Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
        Version: "2012-10-17"
      ManagedPolicyArns:
        - Fn::Join:
            - ""
            - - "arn:"
              - Ref: AWS::Partition
              - :iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
    Metadata:
      aws:cdk:path: CdktestStackNew/Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/ServiceRole/Resource
  CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRoleDefaultPolicy88902FDF:
    Type: AWS::IAM::Policy
    Properties:
      PolicyDocument:
        Statement:
          - Action:
              - s3:GetBucket*
              - s3:GetObject*
              - s3:List*
            Effect: Allow
            Resource:
              - arn:aws:s3:::cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
              - arn:aws:s3:::cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2/*
          - Action:
              - s3:Abort*
              - s3:DeleteObject*
              - s3:GetBucket*
              - s3:GetObject*
              - s3:List*
              - s3:PutObject
              - s3:PutObjectLegalHold
              - s3:PutObjectRetention
              - s3:PutObjectTagging
              - s3:PutObjectVersionTagging
            Effect: Allow
            Resource:
              - arn:aws:s3:::some-testbucket
              - arn:aws:s3:::some-testbucket/*
        Version: "2012-10-17"
      PolicyName: CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRoleDefaultPolicy88902FDF
      Roles:
        - Ref: CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRole89A01265
    Metadata:
      aws:cdk:path: CdktestStackNew/Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/ServiceRole/DefaultPolicy/Resource
  CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C81C01536:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket: cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
        S3Key: c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762.zip
      Environment:
        Variables:
          AWS_CA_BUNDLE: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
      Handler: index.handler
      Layers:
        - Ref: DeployFiles1AwsCliLayer15AA725E
      Role:
        Fn::GetAtt:
          - CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRole89A01265
          - Arn
      Runtime: python3.11
      Timeout: 900
    DependsOn:
      - CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRoleDefaultPolicy88902FDF
      - CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756CServiceRole89A01265
    Metadata:
      aws:cdk:path: CdktestStackNew/Custom::CDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C/Resource
      aws:asset:path: asset.c6358465bf49dfae556bb430bf9c81fa578c221b82c308e3707901b1dd654762
      aws:asset:is-bundled: false
      aws:asset:property: Code
  DeployFiles2AwsCliLayer26C4A816:
    Type: AWS::Lambda::LayerVersion
    Properties:
      Content:
        S3Bucket: cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
        S3Key: 09ee37fb9a746a9dea0618b3024913848346b4b63d08d3fae387b9a11ac83a01.zip
      Description: /opt/awscli/aws
    Metadata:
      aws:cdk:path: CdktestStackNew/DeployFiles2/AwsCliLayer/Resource
      aws:asset:path: asset.09ee37fb9a746a9dea0618b3024913848346b4b63d08d3fae387b9a11ac83a01.zip
      aws:asset:is-bundled: false
      aws:asset:property: Content
  DeployFiles2CustomResource9F1D5822:
    Type: Custom::CDKBucketDeployment
    Properties:
      ServiceToken:
        Fn::GetAtt:
          - CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C81C01536
          - Arn
      SourceBucketNames:
        - cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
      SourceObjectKeys:
        - 7b5d84da957656926886284f7041c1f7e44d118715f664d32787b3075603904e.zip
      DestinationBucketName: some-testbucket
      Prune: true
      OutputObjectKeys: true
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: CdktestStackNew/DeployFiles2/CustomResource/Default
  DeployFiles3AwsCliLayerFEF365E2:
    Type: AWS::Lambda::LayerVersion
    Properties:
      Content:
        S3Bucket: cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
        S3Key: 09ee37fb9a746a9dea0618b3024913848346b4b63d08d3fae387b9a11ac83a01.zip
      Description: /opt/awscli/aws
    Metadata:
      aws:cdk:path: CdktestStackNew/DeployFiles3/AwsCliLayer/Resource
      aws:asset:path: asset.09ee37fb9a746a9dea0618b3024913848346b4b63d08d3fae387b9a11ac83a01.zip
      aws:asset:is-bundled: false
      aws:asset:property: Content
  DeployFiles3CustomResource2CA8E475:
    Type: Custom::CDKBucketDeployment
    Properties:
      ServiceToken:
        Fn::GetAtt:
          - CustomCDKBucketDeployment8693BB64968944B69AAFB0CC9EB8756C81C01536
          - Arn
      SourceBucketNames:
        - cdk-hnb659fds-assets-<<ACCOUNT-ID>>-us-east-2
      SourceObjectKeys:
        - e669d45a003ba25cde82fd486966d8194d784f2a3b8ec48e526255b9c5950432.zip
      DestinationBucketName: some-testbucket
      Prune: true
      OutputObjectKeys: true
    UpdateReplacePolicy: Delete
    DeletionPolicy: Delete
    Metadata:
      aws:cdk:path: CdktestStackNew/DeployFiles3/CustomResource/Default
  CDKMetadata:
    Type: AWS::CDK::Metadata
    Properties:
      Analytics: v2:deflate64:H4sIAAAAAAAA/z2Oy26DQAxFvyX7wc1LbbcJVVddRCB1i5zBQQ7zqPBQhEb8ezVQWN3jY8nXRzi8vcJ+h4Nkum4zw3eIZUDdKhykinKCa69bClcUUnKqYk0/xo+WXPjffGxCoQgFgUuKSRm09xoh5g/3hSN139QJe6dKdo2h4N1n73RIZoP8sclJMVqIhTeU9Jw3b1iPaVxoWksqkwoqHEQbhssgueG5U82/lAEbdo3KewneFiS+7/RyduUVJuV8TfCUl9/DOxz3cN49hTnrehfYEhRL/gFVFpQiOAEAAA==
    Metadata:
      aws:cdk:path: CdktestStackNew/CDKMetadata/Default
Parameters:
  BootstrapVersion:
    Type: AWS::SSM::Parameter::Value<String>
    Default: /cdk-bootstrap/hnb659fds/version
    Description: Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]

Notice that ONLY layer DeployFiles1AwsCliLayer15AA725E is used.

@wimlewis-amazon Feel free to contribute PR to fix the issue.

…tical, unused AwsCliLayers (aws#32907) Created a utility method AwsCliLayer.getOrCreate() for getting or creating the singleton layer construct; updated callers to use it. Fixes aws#32907

…structs create multiple identical AwsCliLayers (aws#32907) Created a utility method AwsCliLayer.getOrCreate() for getting or creating the singleton layer construct; updated callers to use it. As a result, all built-in constructs needing this layer will share a single construct at the root of the stack. Fixes aws#32907

ashishdhingra added bug This issue is a bug. p2 @aws-cdk/aws-s3 Related to Amazon S3 needs-reproduction This issue needs reproduction. labels Jan 21, 2025

ashishdhingra self-assigned this Jan 21, 2025

ashishdhingra added feature-request A feature should be added or improved. and removed bug This issue is a bug. labels Jan 21, 2025

ashishdhingra added bug This issue is a bug. effort/medium Medium work item – several days of effort and removed feature-request A feature should be added or improved. needs-reproduction This issue needs reproduction. labels Jan 21, 2025

ashishdhingra removed their assignment Jan 21, 2025

wimlewis-amazon linked a pull request Jan 23, 2025 that will close this issue

fix(lambda-layer-awscli): s3-deployment, stepfunctions-tasks, eks constructs create multiple identical AwsCliLayers (#32907) #33085

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BucketDeployment creates multiple identical, unused AwsCliLayers #32907

BucketDeployment creates multiple identical, unused AwsCliLayers #32907

wimlewis-amazon commented Jan 14, 2025

ashishdhingra commented Jan 21, 2025

ashishdhingra commented Jan 21, 2025 •

edited

Loading

BucketDeployment creates multiple identical, unused AwsCliLayers #32907

BucketDeployment creates multiple identical, unused AwsCliLayers #32907

Comments

wimlewis-amazon commented Jan 14, 2025

ashishdhingra commented Jan 21, 2025

ashishdhingra commented Jan 21, 2025 • edited Loading

ashishdhingra commented Jan 21, 2025 •

edited

Loading