How to install AWS Load Balancer Controller in a CDK project

Question

I am trying to write a high-level CDK construct that can be used to deploy Django applications with EKS. I have most of the k8s manifests defined for the application, but I am struggling with the Ingress part. Looking into different options, I have decided to try installing the AWS Load Balancer Controller (https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/). Their documentation has instructions for installing the controller using the AWS CLI and the eksctl CLI tool, so I'm working on trying to translate these into CDK code. Here's what I have so far:

import * as ec2 from '@aws-cdk/aws-ec2';
import * as eks from '@aws-cdk/aws-eks';
import * as iam from '@aws-cdk/aws-iam';
import * as cdk from '@aws-cdk/core';
import { ApplicationVpc } from './vpc';
var request = require('sync-request');


export interface DjangoEksProps {
  readonly vpc: ec2.IVpc;
}


export class DjangoEks extends cdk.Construct {

  public vpc: ec2.IVpc;
  public cluster: eks.Cluster;

  constructor(scope: cdk.Construct, id: string, props: DjangoEksProps) {
    super(scope, id);

    this.vpc = props.vpc;


    // allow all account users to assume this role in order to admin the cluster
    const mastersRole = new iam.Role(this, 'AdminRole', {
      assumedBy: new iam.AccountRootPrincipal(),
    });

    this.cluster = new eks.Cluster(this, "MyEksCluster", {
      version: eks.KubernetesVersion.V1_19,
      vpc: this.vpc,
      mastersRole,
      defaultCapacity: 2,
    });

    // Adopted from comments in this issue: https://github.com/aws/aws-cdk/issues/8836
    const albServiceAccount = this.cluster.addServiceAccount('aws-alb-ingress-controller-sa', {
      name: 'aws-load-balancer-controller',
      namespace: 'kube-system',
    });

    const awsAlbControllerPolicyUrl = 'https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json';
    const policyJson = request('GET', awsAlbControllerPolicyUrl).getBody('utf8');
    ((JSON.parse(policyJson))['Statement'] as any[]).forEach(statement => {
      albServiceAccount.addToPrincipalPolicy(iam.PolicyStatement.fromJson(statement))
    })

    // This is where I am stuck
    // https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/deploy/installation/#add-controller-to-cluster

    // I tried running this
    // kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

    this.cluster.addHelmChart('aws-load-balancer-controller-helm-chart', {
      repository: 'https://aws.github.io/eks-charts',
      chart: 'eks/aws-load-balancer-controller',
      release: 'aws-load-balancer-controller',
      version: '2.2.0',
      namespace: 'kube-system',
      values: {
        clusterName: this.cluster.clusterName,
        serviceAccount: {
          create: false,
          name: 'aws-load-balancer-controller',
        },
      },
    });
  }
}

Here are the errors I am seeing in CDK when I do cdk deploy:

Received response status [FAILED] from custom resource. Message returned: Error: b'WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /tmp/kubeconfig\nWARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /tmp/kubeconfig\nRelease "aws-load-balancer-controller" does not exist. Installing it now.\nError: chart "eks/aws-load-balancer-controller" version "2.2.0" not found in https://aws.github.io/eks-charts repository\n' Logs: /aws/lambda/DjangoEks-awscdkawseksKubectlProvi-Handler886CB40B-6yld0A8rw9hp at invokeUserFunction (/var/task/framework.js:95:19) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async onEvent (/var/task/framework.js:19:27) at async Runtime.handler (/var/task/cfn-response.js:48:13) (RequestId: ec066bb2-4cc1-48f6-8a88-c6062c27ed0f)

and a related error:

Received response status [FAILED] from custom resource. Message returned: Error: b'error: no objects passed to create\n' Logs: /aws/lambda/DjangoEks-awscdkawseksKubectlProvi-Handler886CB40B-6yld0A8rw9hp at invokeUserFunction (/var/task/framework.js:95:19) at processTicksAndRejections (internal/process/task_queues.js:93:5) at async onEvent (/var/task/framework.js:19:27) at async Runtime.handler (/var/task/cfn-response.js:48:13) (RequestId: fe2c4c04-4de9-4a71-b18a-ab5bc91d180a)

The CDK EKS docs say that addHelmChart will install the provided Helm Chart with helm upgrade --install.

The AWS Load Balancer Controller installation instructions also say:

Install the TargetGroupBinding CRDs if upgrading the chart via helm upgrade.

kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master"

I'm not sure how I can do this part in CDK. The link to github.com/aws/eks-charts/stable/aws-load-balancer-controller//crds?ref=master gives a 404, but that command does work when I run it against my EKS cluster, and I can install those CRDs. Running the deploy command after manually installing those CRDs also fails with the same message.

I think the error in my code comes from the HelmChartOptions that I pass to the addHelmChart command, and I have tried several options, and referenced similar CDK projects that install Helm charts from the same repo, but I keep getting failures.

Is anyone else installing the AWS Load Balancer Controller with CDK like I am trying to do here? One project that I have been trying to reference is https://github.com/neilkuan/cdk8s-aws-load-balancer-controller.

There is also discussion in this GitHub issue: https://github.com/aws/aws-cdk/issues/8836 that might help as well, but a lot of the discussion is around cert manager manager which doesn't seem to be relevant for what I'm trying to do.

Answer 1

I got some help on the CDK.dev slack channel. I had the wrong version. It should be 1.2.0, the version of the Helm Chart. 2.2.0 is the version of the Controller.

Here's an official example of how to do this from the aws-samples GitHub repo: https://github.com/aws-samples/nexus-oss-on-aws/blob/d3a092d72041b65ca1c09d174818b513594d3e11/src/lib/sonatype-nexus3-stack.ts#L207-L242

Answer 2

There's an open issue for adding kubectl apply -k support https://github.com/aws/aws-cdk/issues/11764