Cloudformation KeyValuePair List as a parameter

Question

When creating ECS infrastructure we describe our Task Definitions with CloudFormation. We want to be able to dynamically pass environment variables as a parameter to the template. According to the docs, Environment has a KeyValuePair type, but CloudFormation parameters do not have this type. We can not hardcode Environment variables to the template, because this template is used as a nested stack so environment variables will be dynamically passed inside it.

The only possible way I see so far is to pass all arguments as a CommaDelimitedList, and then somehow parse and map it using CloudFormation functions. I can Fn::Split every entity in key and value, but how to dynamically build an array of KeyValuePair in CloudFormation?

Or maybe there is an easier way, and I'm missing something? Thanks in advance for any ideas.

Answer 1

I know it's late and you have already found a workaround. However, the following is the closest I came to solve this. Still not completely dynamic as expected parameters have to be defined as placeholders. Therefore the maximum number of environment variables expected should be known.

The answer is based on this blog. All credits to the author.

Parameters:
  EnvVar1:
    Type: String
    Description: A possible environment variable to be passed on to the container definition.
      Should be a key-value pair combined with a ':'. E.g. 'envkey:envval'
    Default: ''
  EnvVar2:
    Type: String
    Description: A possible environment variable to be passed on to the container definition.
      Should be a key-value pair combined with a ':'. E.g. 'envkey:envval'
    Default: ''
  EnvVar3:
    Type: String
    Description: A possible environment variable to be passed on to the container definition.
      Should be a key-value pair combined with a ':'. E.g. 'envkey:envval'
    Default: ''
Conditions:
  Env1Exist: !Not [ !Equals [!Ref EnvVar1, '']]
  Env2Exist: !Not [ !Equals [!Ref EnvVar2, '']]
  Env3Exist: !Not [ !Equals [!Ref EnvVar3, '']]
Resources:
  TaskDefinition:
    ContainerDefinitions:
      -
         Environment:
           - !If
             - Env1Exist
             -
               Name: !Select [0, !Split [":", !Ref EnvVar1]]
               Value: !Select [1, !Split [":", !Ref EnvVar1]]
             - !Ref "AWS::NoValue"
           - !If
             - Env2Exist
             -
               Name: !Select [0, !Split [":", !Ref EnvVar2]]
               Value: !Select [1, !Split [":", !Ref EnvVar2]]
             - !Ref "AWS::NoValue"
           - !If
             - Env3Exist
             -
               Name: !Select [0, !Split [":", !Ref EnvVar3]]
               Value: !Select [1, !Split [":", !Ref EnvVar3]]
             - !Ref "AWS::NoValue"

Answer 2

You may want to consider using the EC2 Parameter Store to create secured key/value pairs, which is supported in CloudFormation, and can be integrated with ECS environments.

AWS Systems Manager Parameter Store

AWS Systems Manager Parameter Store provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, and license codes as parameter values. You can store values as plain text or encrypted data. You can then reference values by using the unique name that you specified when you created the parameter. Highly scalable, available, and durable, Parameter Store is backed by the AWS Cloud. Parameter Store is offered at no additional charge.

While Parameter Store has great security features for storing application secrets, it can also be used to store nonsensitive application strings such as public keys, environment settings, license codes, etc.

And it is supported directly by CloudFormation, allowing you to easily capture, store and manage application configuration strings which can be accessed by ECS. This template allows you provide the Parameter store key values at stack creation time via the console or CLI:

Description: Simple SSM parameter example
Parameters:
  pSMTPServer:
    Description: SMTP Server URL eg [email-smtp.us-east-1.amazonaws.com]:587
    Type: String
    NoEcho: false
  SMTPServer:
    Type: AWS::SSM::Parameter
    Properties: 
      Name: my-smtp-server
      Type: String
      Value: !Ref pSMTPServer

Any AWS runtime environment (EC2, ECS, Lambda) can easily securely retrieve the values. From the console side, there is great Parameter manager interface that maintains parameter version history. Its intergated with IAM, so permissions are controlled with standard IAM policy syntax:

{
    "Action": [
        "ssm:GetParameterHistory",
        "ssm:GetParameter",
        "ssm:GetParameters",
        "ssm:GetParametersByPath"
    ],
    "Resource": [
        "arn:aws:ssm:us-west-2:555513456471:parameter/smtp-server"
    ],
    "Effect": "Allow"
},
{
    "Action": [
        "kms:Decrypt"
    ],
    "Resource": [
        "arn:aws:kms:us-west-2:555513456471:key/36235f94-19b5-4649-84e0-978f52242aa0a"
    ],
    "Effect": "Allow"
}

Finally, this blog article shows a technique to read the permissions into a Dockerfile at runtime. They suggest a secure way to handle environment variables in Docker with AWS Parameter Store. For reference, I am including their Dockerfile here:

FROM grafana/grafana:master

RUN curl -L -o /bin/aws-env https://github.com/Droplr/aws-env/raw/master/bin/aws-env-linux-amd64 && \
  chmod +x /bin/aws-env

ENTRYPOINT ["/bin/bash", "-c", "eval $(/bin/aws-env) && /run.sh"]

With that invocation, each of the parameters are available as an environment variable in the container. You app may or may not need a wrapper to read the parameters from the environment variables.

Answer 3

I was facing the same problem ,I needed to create a lambda resource with environment variables. We decided to fix initial set of environment variable and keys name are also decided in advance. So I added four parameters , and used Ref for values while keeping fixed keys name.

Answer 4

There is another way too - which may sound overkill but it allows to put whatever env. to the function you wish, no need to "predefine" how many env. variables, only restriction in sample below - can not use :::: or |||| inside value of the key. Key can't have such symbols by AWS docs already.

Gameplan:

Make an inline CF Lambda Function with code which accepts all env in any format you wish as a string and uses any code you want to use inside that function (i use JS with NodeJS env) and while it's your code, parse how you wish that string passing in and use aws-sdk to update the function. Call function once inside the CF template.

In this sample you pass in env as such string:

key1::::value1||||key2::::value2 If you need to use :::: or |||| in your value, of course update to some other divider.

Not big fan of running lambda for such task, yet i want to have option of passing in virtually any env. to the CF template and this works.

  LambdaToSetEnvRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchLambdaInsightsExecutionRolePolicy
      Policies:
        - PolicyName: cloudwatch-logs
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource:
                  - !Sub "arn:aws:logs:*:${AWS::AccountId}:log-group:*:*"
                  - !Sub "arn:aws:logs:*:${AWS::AccountId}:log-group:/aws/lambda-insights:*"
        - PolicyName: trigger-lambda-by-cloud-events
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - 'lambda:UpdateFunctionConfiguration'
                Resource:
                  - !GetAtt OriginalLambda.Arn
      Tags:
        - { Key: managed-by, Value: !Ref AWS::StackId }
  LambdaToSetEnv:
    Type: AWS::Lambda::Function
    DeletionPolicy: Delete
    Properties:
      Code:
        ZipFile: |
          const response = require('cfn-response');
          const aws = require('aws-sdk');
          exports.handler = (event, context) => {
            console.log(JSON.stringify({event, context}));
            try {
              if (event.RequestType === "Delete") {
                response.send(event, context, response.SUCCESS, {RequestType: event.RequestType});
              } else {
                const client = new aws.Lambda({apiVersion: '2015-03-31'});
                const Variables = {
                  "All": process.env.FunctionEnvVariables,
                };
                console.log('process.env.FunctionEnvVariables: ', process.env.FunctionEnvVariables);
                if(process.env.FunctionEnvVariables){
                  process.env.FunctionEnvVariables.split('||||').forEach((pair) => {
                    if(pair && pair.trim() !== ''){
                      Variables[pair.split('::::')[0]] = pair.split('::::')[1];
                    }
                  })
                }
                const result = client.updateFunctionConfiguration({ FunctionName: process.env.LambdaToUpdateArn, Environment: { Variables } }, function (error, data){
                  console.log('data: ', data);
                  console.log('error: ', error);
                  if(error){
                    console.error(error);
                    response.send(event, context, response.ERROR, {});      
                  } else {
                    response.send(event, context, response.SUCCESS, {});
                  }
                });
              }
            } catch (e) {
              response.send(event, context, response.ERROR, e.stack);
            }
          }
      Role: !GetAtt LambdaToSetEnvRole.Arn
      Handler: index.handler
      Runtime: nodejs14.x
      Timeout: '300'
      Environment:
        Variables:
          LambdaToUpdateArn: !GetAtt OriginalLambda.Arn
          FunctionEnvVariables: !Ref FunctionEnvVariables
  LambdaCall:
    DependsOn:
      - OriginalLambda
      - LambdaToSetEnv
    Type: Custom::LambdaCallout
    Properties:
      ServiceToken: !GetAtt LambdaToSetEnv.Arn