CDK deploy results in "Bucket named 'x' exists, but not in account 067685711111. Wrong account?"

Question

I am modifying a CDK pipeline that works. I am adding a lambda, and when this code modification is made, the pipeline fails with an error.

import * as cdk from '@aws-cdk/core';
import * as lambda from '@aws-cdk/aws-lambda';
import * as path from 'path';
import { Rule, Schedule } from '@aws-cdk/aws-events';
import { LambdaFunction } from '@aws-cdk/aws-events-targets';
import { Secret } from '@aws-cdk/aws-secretsmanager';
import { Effect, Policy, PolicyStatement } from '@aws-cdk/aws-iam';

interface LambdaStackProps extends cdk.StackProps {
    logzioToken: string;
}

export class Lambda extends cdk.Stack {
    constructor(scope: cdk.Construct, id: string, props: LambdaStackProps) {
        super(scope, id, props);

        const logzioSecret = Secret.fromSecretAttributes(this, 'sv-logzioToken', {
            secretPartialArn: props.logzioToken,
        });

        // Here is the func itself
        const func = new lambda.Function(this, 'sv-http-cache-clear-lambda', {
            runtime: lambda.Runtime.NODEJS_14_X,
            handler: 'index.handler',
            // The "build" folder requires `yarn build` to have been run in the lambda folder
            code: lambda.Code.fromAsset(path.join(__dirname, '../../lambdas/HttpCacheClear/build')),
            environment: {
                LOGZIO_TOKEN: logzioSecret.secretValueFromJson('logzioToken').toString(),
            },
        });

        // Here are the Dynamo permissions I think we need
        const policyStatement = new PolicyStatement({
            actions: ['dynamodb:GetItem', 'dynamodb:Query', 'dynamodb:Scan', 'dynamodb:DeleteItem'],
            effect: Effect.ALLOW,
            resources: ['*'],
        });
        const policy = new Policy(this, 'sv-http-cache-clear-policy', {
            statements: [policyStatement],
        });
        func.role?.attachInlinePolicy(policy);

        new Rule(this, 'sv-http-cache-clear-schedule-rule', {
            description: 'Schedule a Lambda that regularly clears the cache',
            schedule: Schedule.cron({
                year: '*',
                month: '*',
                day: '*',
                hour: '1',
                minute: '0',
            }),
            targets: [new LambdaFunction(func)],
        });
    }
}

This is instantiated in the CDK from here:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '@aws-cdk/core';
import { AppStack } from '../lib/app-stack';
import { ECR } from '../lib/ecr';
import { Lambda } from '../lib/lambda';
import { getConfig } from '../lib/config';

const app = new cdk.App();

// Stateful infra
const { repository } = new ECR(app, 'sv-ecr-stack');

// Get app config
const environment = process.env.ENVIRONMENT || '';
const { certificateArn, domainName, sIdProgramId, logzioToken, optimizelyUrl } = getConfig(environment);

new AppStack(app, 'sv-stack', {
    repository,
    domainName,
    certificateArn,
    sIdProgramId,
    logzioToken,
    optimizelyUrl,
    environment,
});

const lambda = new Lambda(app, 'sv-http-scheduled-cache-clear', {
    logzioToken,
});

For reasons of brevity I have not added the AppStack to this post.

Finally I have a fairly simple set of NPM dependencies:

{
    "name": "cdk",
    "version": "0.1.0",
    "bin": {
        "cdk": "bin/cdk.js"
    },
    "scripts": {
        "build": "tsc",
        "watch": "tsc -w",
        "cdk": "cdk",
        "synth": "cdk synth -q",
        "deploy": "cdk deploy --all --progress=events --require-approval=never"
    },
    "devDependencies": {
        "@aws-cdk/assert": "^1.100.0",
        "@aws-cdk/aws-certificatemanager": "^1.100.0",
        "@aws-cdk/aws-ec2": "^1.100.0",
        "@aws-cdk/aws-ecr": "^1.100.0",
        "@aws-cdk/aws-ecs": "^1.100.0",
        "@aws-cdk/aws-ecs-patterns": "^1.100.0",
        "@aws-cdk/aws-iam": "^1.100.0",
        "@aws-cdk/aws-route53": "^1.100.0",
        "@types/node": "10.17.27",
        "aws-cdk": "^1.100.0",
        "ts-node": "^9.0.0",
        "typescript": "~3.9.7"
    },
    "dependencies": {
        "@aws-cdk/core": "^1.100.0",
        "source-map-support": "^0.5.16"
    }
}

If I run npm run synth locally then it runs the synth command, and succeeds. I have no TypeScript errors in my editor.

Here is the cdk deploy error:

export AWS_ACCESS_KEY_ID=$DEV_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY=$DEV_SECRET_ACCESS_KEY
export ENVIRONMENT=dev
cd cdk
npm install
npm run deploy

npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

added 1363 packages from 232 contributors and audited 1366 packages in 19.228s

18 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities


> [email protected] deploy /home/circleci/project/cdk
> cdk deploy --all --progress=events --require-approval=never

sv-ecr-stack
sv-ecr-stack: deploying...

 ✅  sv-ecr-stack (no changes)

Outputs:
sv-ecr-stack.ExportsOutputFnGetAttsvrepositoryXXX = arn:aws:ecr:*********:067685711111:repository/student-validation-app
sv-ecr-stack.ExportsOutputRefsvrepositoryXXX = student-validation-app

Stack ARN:
arn:aws:cloudformation:*********:067685711111:stack/sv-ecr-stack/aabbccddeeffgghhiijjkk
sv-http-scheduled-cache-clear
sv-http-scheduled-cache-clear: deploying...
[0%] start: Publishing abcdefghijklmnop:current
[100%] fail: Bucket named 'cdktoolkit-stagingbucket-7k57fjeur592' exists, but not in account 067685711111. Wrong account?

 ❌  sv-http-scheduled-cache-clear failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at Object.publishAssets (/home/circleci/project/cdk/node_modules/aws-cdk/lib/util/asset-publishing.ts:25:11)
    at Object.deployStack (/home/circleci/project/cdk/node_modules/aws-cdk/lib/api/deploy-stack.ts:235:3)
    at CdkToolkit.deploy (/home/circleci/project/cdk/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
    at initCommandLine (/home/circleci/project/cdk/node_modules/aws-cdk/bin/cdk.ts:208:9)
***************************************************
*** Newer version of CDK is available [1.110.0] ***
*** Upgrade recommended                         ***
***************************************************
Failed to publish one or more assets. See the error messages above for more information.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] deploy: `cdk deploy --all --progress=events --require-approval=never`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] deploy script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/circleci/.npm/_logs/2021-06-25T13_34_37_706Z-debug.log


Exited with code exit status 1

A bit of research indicates that CDK uses S3 to create a "staging bucket", and I assume that is what the cdktoolkit-stagingbucket-* is (it is not an artifact that I have created explicitly).

Do I need to create this bucket remotely/explicitly or locally/manually?

Update

I have found this bug report which suggests that the error message is terrible - it needs S3 acccess but the message does not make that clear. I have found this policy file for the CI/CD user, and I wonder if I need to work out what policy to add to it to permit S3 usage.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "StmtXXXYYYZZZ4501",
            "Action": [
                "cloudformation:*",
                "ec2:*",
                "ecs:*",
                "ecr:*",
                "application-autoscaling:*",
                "elasticloadbalancing:*",
                "iam:CreateRole",
                "iam:DeleteRole",
                "iam:GetRole",
                "iam:TagRole",
                "iam:UpdateRole",
                "iam:GetRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:PutRolePolicy",
                "iam:PassRole",
                "logs:*",
                "route53:*",
                "secretsmanager:*",
                "sts:*",
                "dynamodb:*",
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Answer 1

The error message is rather misleading. The S3 bucket is highly likely to exist in another account if you have access to several accounts - this is because it is an auto-generated bucket that is used by CDK to package and deploy lambdas. I assume it is the same name in all AWS accounts.

The simple solution is to add "s3:*" to your allowed actions against a "*" resource. However, I decided to add a new Policy so that the resource specification was a little tighter. In my case this meant adding a new clause with a different resource specification:

{
    "Action": [
        "s3:*"
    ],
    "Effect": "Allow",
    "Resource": "arn:aws:s3:::cdktoolkit-stagingbucket-*"
}

It looks like you don't have to explicitly create this bucket - CDK will do it for you. It appears also that the "Sid" is not a mandatory field.

In my case, adding this Policy revealed how many other permissions I was missing, but at least the error message for those was more sensible!

Answer 2

The action that triggers that error is s3:GetBucketLocation. A policy that only provides minimal permissions the CDK needs would be:

{
    "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:PutObject",
        "s3:DeleteObject"
    ],
    "Resource": "arn:aws:s3:::cdktoolkit-stagingbucket-*",
    "Effect": "Allow"
}