Lambda can't find modules from outer folders when deployed with CDK

Question

I am deploying some apis to API Gateway using cdk. My problem is the file that contains the lambda(index.ts) can't import any files or npm modules outside that folder(folder named get-users).

I tried copying node_modules folder and other files (which were outside the folder get-users) to the folder get-users and it worked perfectly.

Example error when importing lodash is as follows,

"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module 'lodash'",
"stack": [
    "Runtime.ImportModuleError: Error: Cannot find module 'lodash'",

I am importing lodash as follows,

import * as _ from "lodash";

I am importing shared files as follows,

import { validator } from "./shared/validators" // This one works

import { validator } from "../../shared/validators" // This one doesn't work

Answer 1

I found the answer after some research. Problem was CDK not deploying the node_modules folder and other folders which are outside the folder which contains the lambda source file.

When creating the lambda file root path has to be added to the 'code' attribute so that it will take all the folders/files inside it and deploy to the lambda.

    const pathToRoot = //absolute path to the root folder
    const pathToHandler = //path to handler from root folder
    const lambdaFunction: Function = new Function(construct, name, {
        functionName: name,
        code: Code.asset(pathToRoot),
        handler: `${pathToHandler}/index.handler`,
        runtime: Runtime.NODEJS_10_X
    });

UPDATE

And after some more research found out the best way to handle this is with Lambda Layers. I have added my node modules folder as a layer.

Answer 2

If your lambda function uses dependencies, you'll have to package everything into a ZIP file and tell CDK where to find the ZIP. It will then upload it for you to S3. But the packaging will be your responsibility. You need to include all the dependencies as well as your code.

CDK currently supports 3 kinds of "Assets":

• InlineCode - useful for one-line-lambdas
• AssetCode - one-file lambdas without dependencies
• S3Code - existing lambda packages already uploaded in an S3 bucket

For your use-case, you'll need AssetCode, but instead of a directory, you'll point to local ZIP file.

Related answer: How to install dependencies of lambda functions upon cdk build with AWS CDK

Since quite a few people ask about this, I'll see if I can open-source my lambda packaging construct in Python. If yes, I'll link it here.

Edit: I opensourced a CDK construct which I use for lambda packaging with more than a hundred dependencies including NumPy.

https://gitlab.com/josef.stach/aws-cdk-lambda-asset

Answer 3

After spending some time contemplating the very same issue I finally resorted to using webpack to build my lambda packages, which is definitely the way I would recommend.

What you would do in that case is to point your AssetCode to a directory where Webpack has previously packaged the whole thing e.g. src/your-function/build.

Additional benefit is that you can configure Webpack to minify the files, so you speed up your deployment, but also your lambda cold start.

Answer 4

Just want to throw in a new answer to this question. In CDK, check out the aws-lambda-nodejs module, which wasn't included in other answers to this question. Is short, it's kind of designed to solve this problem. For example, putting all the other functionality it brings to one side, by simply using this contstruct:

  const testLambda = new lambda_nodejs.NodejsFunction(this, 'fox-get', {
    runtime: lambda.Runtime.NODEJS_12_X,
    entry: 'lambda/fox-get/fox-get.js',
    handler: 'handler',
    functionName: 'sn-v1-fox-get',
    description: 'Get all devices from fox API.',
    memorySize: 256,
    timeout: Duration.seconds(360)
  });

it will fire up docker and produce a nice minified single file. It also allows you to work with node modules inside the project. I haven't done this, but lots of other developers have and you can easily google what they've done.

This was enough for me. I can develop my lambda functions inside a clean folder tree, and avoid duplicate source files, which was all I needed!

Answer 5

I ran into a similar issue. I was trying to access a function inside the grandparent folder of where the lambda was defined. The problem was that the lambda was only able to access files in the same folder (it couldn't access anything in a directory up). For example the below lambda snippet was not working:

import json
from ....function_outside_of_cdk_dir import outer_fn_hello

def handler(event, context):
    print('EVENT RECEIVED!!!!!!!!: {}'.format(json.dumps(event)))
    outer_fn_hello("test")

To solve this, you can use DockerImageFunction. The solution is putting all of the files your lambda will need (including a file from the grandparent folder) onto a docker container and then running your lambda from that docker container. Below are some screenshots and code that hopefully will help someone (in Python):

We have a directory structure like below:

And inside outer-folder > cdk-example:

function_outside_of_cdk_dir.py

def outer_fn_hello(name):
    print("OUTER FN HELLO", name)

hello_lambda.py (note how it can access the function_outside_of_cdk_dir.py)

import json
from function_outside_of_cdk_dir import outer_fn_hello

def handler(event, context):
    print('EVENT RECEIVED!!!!!!!!: {}'.format(json.dumps(event)))
    outer_fn_hello("test")

Dockerfile

FROM public.ecr.aws/lambda/python:3.8
ARG APP_ROOT=.
COPY ${APP_ROOT}/function_outside_of_cdk_dir.py ${LAMBDA_TASK_ROOT}/
COPY ${APP_ROOT}/cdk-example/functions ${LAMBDA_TASK_ROOT}/functions


ENV PYTHONPATH=${LAMBDA_TASK_ROOT}

ENTRYPOINT ["/lambda-entrypoint.sh"]
CMD ["functions/hello_lambda.handler"]

cdk-example --> cdk_example --> cdk_example_stack.py

from aws_cdk import core as cdk
import os
from aws_cdk.aws_lambda import DockerImageFunction, DockerImageCode
from aws_cdk import core

class CdkExampleStack(cdk.Stack):

    def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
        docker_context = os.path.join(os.path.dirname(__file__), '../../')

        # The code that defines your stack goes here

        #define lambda
        lambda_1 = DockerImageFunction(self, 'lambda_1', timeout=cdk.Duration.seconds(30),  function_name=f'lambda_1', code=DockerImageCode.from_image_asset(
            docker_context, file='cdk-example/functions/Dockerfile'))

.dockerignore

cdk-example/cdk.out/

Answer 6

I had a similar issue with a brand new CDK app. I specified my function with:

const handler = new lambda.Function(this, "RoutesHandler", {
      runtime: lambda.Runtime.NODEJS_14_X,
      code: lambda.Code.fromAsset("resources"),
      handler: "handler.main",
      environment: {
        BUCKET: bucket.bucketName
      }
});

My mistake

My mistake was I had put a file /resources/main.ts instead of /resources/main.js