life is too short for a diary

Creating Python AWS lambda layer with Docker

Tags: docker aws python

When you develop AWS lambda functions, you might feel the need to install additional python libraries. This can be achieved using Lambda layers that can be included in any lambda function.

Creating Lambda Layer with Docker

We will create a lambda layer my-lambda-layer.

$ mkdir my-lambda-layer
$ cd my-lambda-layer

Next, we need to list libraries that we want to include in the lambda layer. For our example, we will only add pandas to our lambda layer.

$ echo "pandas" > requirements.txt

Use docker to install libraries

$ mkdir -p python/lib/python3.7/site-packages
$ docker run \
  -v "$(pwd):/var/task" \
  "lambci/lambda:build-python3.7" \
  /bin/sh -c "pip install -r requirements.txt \
  -t python/lib/python3.7/site-packages/; exit"

This will create lambda layer for python runtime 3.7. However you can use a different runtime by choosing a different version like 3.6, 3.8, etc

The example directory structure for a Lambda layer that’s compatible with Python 3.7

├── requirements.txt
└── python/
    └── lib/
        ├── python3.7/
        │   └── site-packages/

Remove unnecessary files (Optional)

Sometimes you might want to reduce the size of the lambda layer since there is a size quota in AWS1.

$ find -name "__pycache__" -type d | xargs rm -rf
$ find ./ -name '*.pyc' -type f -delete
$ find -name "tests" -type d | xargs rm -rf

Create a zipped file

$ chmod -R 777 python/
$ zip -r python

Create or update your layer

$ aws lambda publish-layer-version --layer-name my-lambda-layer --description "Demo Lambda Layer" --zip-file "fileb://" --compatible-runtimes "python3.7"


comments powered by Disqus