Creating MSK event source mapping for Lambda function fails

Question

I am using the AWS-CDK to create a stack with an AWS-MSK cluster and a Lambda function which should be triggered, when a new message is available in a specific topic.

I already had it working nicely and then I decided to add clientAuthentication and now I am stuck. I am using SASL/SCRAM for authentication. I have created a custom encryption key via the KMS service and I am using that key in a Secret in the SecretsManager. I have associated that Secret with my MSK cluster and turned on clientAuthentication there.

I have also already created an interface endpoint in my VPC to the Lambda Service in order for the service to be able to access my cluster (again, this already worked when I hadn't activated clientAuthentication).

Now I am defining my Lambda listener handler function like this:

const listener = new aws_lambda.Function(this, 'ListenerHandler', {
  vpc,
  vpcSubnets: { subnetGroupName: 'ListenerPrivate' },
  runtime: aws_lambda.Runtime.NODEJS_14_X,
  code: aws_lambda.Code.fromAsset('lambda'),
  handler: 'listener.handler'
});

listener.addToRolePolicy(new aws_iam.PolicyStatement({
  effect: Effect.ALLOW,
  actions: ['kafka:*', 'kafka-cluster:*', 'secretsmanager:DescribeSecret', 'secretsmanager:GetSecretValue'],
  resources: [cluster.ref]
}));

const secretsFromLambdaAccessRole = new aws_iam.Role(this, 'AccessSecretsFromLambdaRoles', {
  assumedBy: new aws_iam.ServicePrincipal('kafka.amazonaws.com')
});
secretsFromLambdaAccessRole.addToPolicy(new aws_iam.PolicyStatement({
  effect: Effect.ALLOW,
  actions: ['secretsmanager:DescribeSecret', 'secretsmanager:GetSecretValue'],
  resources: [KAFKA_ACCESS_SECRET_ARN]
}));

listener.role?.addManagedPolicy(
    aws_iam.ManagedPolicy
        .fromAwsManagedPolicyName("service-role/AWSLambdaVPCAccessExecutionRole")
);

listener.role?.addManagedPolicy(
    aws_iam.ManagedPolicy
        .fromAwsManagedPolicyName("service-role/AWSLambdaMSKExecutionRole")
);

const kafkaAccessSecret = aws_secretsmanager.Secret
    .fromSecretCompleteArn(this, 'kafkaAccessSecret', KAFKA_ACCESS_SECRET_ARN);

listener.addEventSource(new ManagedKafkaEventSource({
  clusterArn: cluster.ref,
  topic: "MyTopic",
  startingPosition: StartingPosition.LATEST,
  secret: kafkaAccessSecret,
}));

The secret also has policies assigned to it:

{
    "Version" : "2012-10-17",
    "Statement" : [ {
        "Sid" : "AWSLambdaResourcePolicy",
        "Effect" : "Allow",
        "Principal" : {
            "Service" : "lambda.amazonaws.com"
        },
        "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:ListSecretVersionIds" ],
        "Resource" : "arn:aws:secretsmanager:some-region:some-account:secret:AmazonMSK_some-secret"
    }, {
        "Sid" : "AWSKafkaResourcePolicy",
        "Effect" : "Allow",
        "Principal" : {
            "Service" : "kafka.amazonaws.com"
        },
        "Action" : "secretsmanager:GetSecretValue",
        "Resource" : "arn:aws:secretsmanager:some-region:some-account:secret:AmazonMSK_some-secret"
    } ]
}

Now, when I try to deploy my lambda function via the CDK and it comes to the point, where it should add the event source mapping, I get this error:

Failed resources:
MskExampleStack | 17:23:22 | CREATE_FAILED        | AWS::Lambda::EventSourceMapping       | ListenerHandler/KafkaEventSource:MskExampleStackListenerHandler4711MyTopic (ListenerHandlerKafkaEventSourceMskExampleStackListenerHandler4711MyTopic0815) 
Resource handler returned message: "Invalid request provided: Cannot access secret manager value arn:aws:secretsmanager:some-region:some-account:secret:AmazonMSK_dev-some-secret.
Please ensure the role can perform the 'secretsmanager:GetSecretValue' action on your broker in IAM. (Service: Lambda, Status Code: 400, Request ID: 123456789, Extended Request ID: null)" (RequestToken: 987654321, HandlerErrorCode: InvalidRequest)

I cannot figure out, what I am missing. What role is the error referring to? Where do I need to add the action "secretsmanager:GetSecretValue"? My user has complete Admin Rights.

Answer 1

You need the following:

1. kms permissions on the lambda role
2. secretsmanager permissions on the lambda role
3. (What I was missing) lambda.amazonaws.com in the key policy for your kms key

my setup:

lambda permissions:

    - Effect: "Allow"
      Action:
        - kms:Decrypt
        - kms:GenerateDataKey*
      Resource:
        - "*"
    - Effect: "Allow"
      Action:
        - secretsmanager:GetSecretValue
      Resource:
        - "your secret arn"

KMS Policy:

      {
            "Sid": "Decrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "lambda.amazonaws.com"
                ]
            },
            "Action": [
                "kms:GenerateDataKey*",
                "kms:Decrypt",
            ],
            "Resource": "*"
        }

Amazon has a knack for writing about 500 needless words to document a feature and never document things around using KMS with that feature even though it seems to vary greatly.