Among the languages supported by AWS Lambda, Python seems to be popular due to its ease of execution and low learning cost. Although my main language is Java, I often find myself wanting to use Python for quick Lambda implementations, even if I'm not very familiar with it. When you want to leverage external libraries in Lambda, you need to prepare something called a layer. If the pre-prepared layers by AWS suffice, that's great, but if not, you need to create your own layer. However, when I tried to do this, I got stuck with Python environment setup (especially related to OpenSSL), which was quite challenging. Therefore, I wanted to avoid the hassle of setting up a Python environment and thought it would be great to use Docker and execute tedious command tasks via a shell, so I created a layer creation tool. After various trials and errors, I decided to run the Docker container of the layer creation tool on AWS Batch. I use CodeBuild for image building and have prepared a CloudFormation template for setting up the CodeBuild and AWS Batch environments.

Before trying AWS Batch for this article, I tried all of the following, but either it was impossible or, even if it was possible, it wasn't satisfactory, so I gave up.

I thought it would be convenient and interesting if I could create a Lambda layer with Lambda itself, so I tried it. When running Lambda in a container, you can only write to the /tmp directory. I set the installation destination to /tmp with pip install, but eventually, the installation management information was written to directories other than /tmp, causing an error, so I gave up.

For reference, the "Container Constraints" section of AWS Lambda for the Containers Developer states the following:

Containers run with a read-only root file system (only the /tmp path is writable).

I thought it was nice because Docker can be used by default and it's free, so I tried it. However, it seems that the CPU architecture of CloudShell itself is only x86_64, and I couldn't run container images created with arm64, which Amazon is promoting, so I gave up. Even after trying to install a cross-platform emulator for Docker, it didn't work.

I prepared an arm64 instance, so naturally, I could run container images created with arm64. However, there were too many steps to use it each time, and I couldn't limit the instance running time purely to the container execution time, which seemed like a waste of cost, so AWS Batch seemed like a good option.

I created a layer creation tool. It can be used anywhere as long as Docker is available, including local development environments.

• PublishPythonLambdaLayer.sh

The general steps to create and register a Python Lambda layer are as follows:

1. Execute the Python library installation command
   • Prepare a requirements.txt file listing the target libraries
   • python -m pip install [options] -r <requirements file> [package-index-options] ...
2. Create the layer (zip file)
   • It must be prepared with the specified directory structure and compressed into a zip file
     • python
     • python/lib/python3.x/site-packages (site directory)
3. Upload the layer (zip file) to S3
   • With AWS CLI, use s3 cp
   • With the AWS Management Console, you can upload the layer zip file directly when creating the layer, but if it exceeds 10MB, you need to place it in S3 for upload
4. Register the Lambda layer
   • With AWS CLI, use publish-layer-version

The layer creation shell performs these steps in one go. Note that it does not support creating Lambda layers that include custom Python modules.

If you do not specify the S3 bucket name for uploading the created Lambda layer, it will only create the layer. This is to allow usage up to layer creation even if you know in advance that the environment you are executing from cannot connect to S3. As long as you can create the layer (zip file) and have the file on hand, you can handle it manually.

Additionally, if you have an environment where Python can be used normally, you can directly use only the layer creation shell to create the layer.

You can check the details of how to use the layer creation shell with the following command:

./PublishPythonLambdaLayer.sh -h

• Dockerfile

# see https://hub.docker.com/r/amazon/aws-lambda-python/
ARG BASE_IMAGE_TAG=latest

FROM amazon/aws-lambda-python:${BASE_IMAGE_TAG?}

COPY ./PublishPythonLambdaLayer.sh .

RUN <<EOF
chmod +x ./PublishPythonLambdaLayer.sh

# for Amazon Linux 2023 (for python 3.12)
if (type dnf > /dev/null 2>&1); then
  # see https://docs.aws.amazon.com/linux/al2023/ug/deterministic-upgrades-usage.html
  # dnf update -y
  dnf upgrade -y --releasever=latest
  dnf install -y zip unzip
  dnf clean all
  rm -rf /var/cache/dnf/*
# for Amazon Linux 2 (for python 3.11)
elif (type yum > /dev/null 2>&1); then
  yum update -y
  yum install -y zip unzip
  yum clean all
  rm -rf /var/cache/yum/*
else
  echo update failed. unsupported package management tool.
  exit 1
fi

# see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
if !(type aws > /dev/null 2>&1); then
  CPU_ARCHITECTURE=$(uname -m)
  if [ "$CPU_ARCHITECTURE" = "x86_64" ]; then
    curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
  elif [ "$CPU_ARCHITECTURE" = "aarch64" ]; then
    curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"
  else
    echo aws cli install failed. unsupported cpu architecture.
    exit 1
  fi
  unzip -qq awscliv2.zip
  ./aws/install > /dev/null 2>&1
  rm -f awscliv2.zip
fi
EOF

ENTRYPOINT [ "bash", "./PublishPythonLambdaLayer.sh" ]
CMD [ "" ]

1. Base Image
   • We use amazon/aws-lambda-python. It's the actual AWS Lambda Python runtime environment, making it the optimal environment for creating layers.
2. Base Image Tag
   • You can specify the tag of the base image during docker build with --build-arg BASE_IMAGE_TAG=<tag>. You can flexibly choose an environment that matches the desired CPU architecture and Python version. For example, you can specify 3.11 or 3.12. However, Python 3.11 based on Amazon Linux 2 only uses yum, and Python 3.12 based on Amazon Linux 2023 only uses dnf, which are not compatible. Therefore, I checked if the dnf command is available and handled it by switching between dnf and yum.
3. Initialization Processes like Installation
   • Written using here documents. It makes it easier to write complex processes without having to chain commands with && \, which in turn reduces the number of image layers, making the image size smaller.
4. Others
   • I cut out the logs of file extraction and installation of AWS CLI because they were overwhelming.
   • The options for the layer creation shell can be overwritten with CMD at runtime.

1. Place the source (layer creation tool) in S3
2. Create a Docker image with CodeBuild
3. Push to ECR

1. Pull the Docker image from ECR with AWS Batch
2. Execute the container with AWS Batch to create the layer
3. Upload to S3
4. Register the layer with Lambda

AWS officially strongly recommends using arm64 for the CPU architecture of computing resources, including Lambda, so the default values in the CloudFormation template for environment creation are set for arm64.

Prepare a subnet that meets all the following network requirements to create an AWS Batch environment. You can easily prepare it using the CloudFormation template from my previous article, "Gently Building a Network with AWS CloudFormation - IPv6 Support, Easy NAT Gateway Setup, and Wallet-Friendly!".

• Able to connect to S3
  • If connecting to an S3 bucket that is not externally accessible, you need to connect via a private link (prepare a Gateway VPC endpoint and route it)
• Able to connect to the internet via IPv4

Since CodeBuild is set to reference the source code from S3, prepare an S3 bucket and folder to store the layer creation tool source code, and store the layer creation tool source code (Dockerfile and PublishPythonLambdaLayer.sh) at the same level. The requirements.txt used later is placed in the same location for convenience, but it can be placed in another bucket.
For example, I prepared s3://publish-lambda-layer-tool-pcjkn63zhk/python/.
It is desirable to check "Block all public access" when creating the S3 bucket.

Prepare an S3 bucket to upload the created Lambda layers.
For example, I prepared s3://lambda-layer-pcjkn63zhk/.
It is desirable to check "Block all public access" when creating the S3 bucket.

I have prepared a CloudFormation template to create a CodeBuild project and ECR repository. Use this to create the resources.

• cfn_codebuild.yaml

Below is an example of parameter settings for arm64.

You can also prepare for x86_64. Below is an example of parameter settings.

Execute the build in the created CodeBuild project. The buildspec, which describes the process of creating a Docker image and pushing it to the ECR repository, and the environment variables referenced within the buildspec are already prepared when the project is created with CloudFormation. You can build it by simply setting the environment variables to be set at runtime.

version: 0.2
phases:
  pre_build:
    commands:
      - |
        export AWS_ECR_REGISTRY_URL=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.$AWS_URL_SUFFIX
        export DOCKER_IMAGE_REPO_NAME_AND_TAG=$IMAGE_REPO_NAME:$IMAGE_TAG
        echo "AWS_ECR_REGISTRY_URL="$AWS_ECR_REGISTRY_URL
        echo "DOCKER_IMAGE_REPO_NAME_AND_TAG="$DOCKER_IMAGE_REPO_NAME_AND_TAG
      - echo "----- login to Amazon ECR Repository -----"
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ECR_REGISTRY_URL
      - |
        if [ -n "$DOCKER_USER" ] && [ -n "$DOCKER_TOKEN" ]; then
          echo "----- login to docker hub -----"
          echo $DOCKER_TOKEN | docker login -u $DOCKER_USER --password-stdin
        fi
  build:
    commands:
      - echo "----- build docker image -----"
      - echo Build started on `date`
      - docker build -t $DOCKER_IMAGE_REPO_NAME_AND_TAG $DOCKER_BUILD_OPTIONS .
      - echo Build completed on `date`
  post_build:
    commands:
      - echo "----- add docker image tag -----"
      - docker tag $DOCKER_IMAGE_REPO_NAME_AND_TAG $AWS_ECR_REGISTRY_URL/$DOCKER_IMAGE_REPO_NAME_AND_TAG
      - echo "----- push docker image to Amazon ECR Repository -----"
      - docker push $AWS_ECR_REGISTRY_URL/$DOCKER_IMAGE_REPO_NAME_AND_TAG

Set them in the CodeBuild project's "Edit" -> "Environment" -> "Additional settings" -> "Environment variables". After setting, press "Update project".

Below is an example of parameter settings for arm64.

Below is an example of parameter settings for x86_64.

Press "Start build".

Confirm that it was successful.

Confirm that the image has been pushed to the ECR repository.

Sometimes, when executing the docker build command, the pull of the base image fails with an error message like "Too Many Requests." If you encounter this, simply retrying the build might succeed. If retries continue to fail, or if you want to ensure stable build success, prepare your own Docker Hub user and set the DOCKER_USER and DOCKER_TOKEN environment variables in CodeBuild. It is recommended to register these authentication details in Parameter Store or Secrets Manager instead of setting them directly as plain text in environment variables.

Parameter Store Example

Secrets Manager Example

CodeBuild Environment Variable Settings Example

• When referring to Parameter Store
  • Set the value to "The name of my parameter" and the type to "Parameter".
    • Example value: DOCKER_USER
• When referring to Secrets Manager
  • Set the value to "Secret name:Secret key" and the type to "Secrets Manager".
    • Example value: MyDockerHubAccount:DOCKER_TOKEN

I have prepared a CloudFormation template to create an AWS Batch environment. Use this to create the resources.

• cfn_aws_batch.yaml

Below is an example of parameter settings for arm64.

Below is an example of parameter settings for x86_64.

Create a job in the created AWS Batch environment to create a Lambda layer. In AWS Batch, go to "Jobs" and press the "Submit new job" button.

Select the following:

• Job Definition
  • CloudFormation parameter "prefix" value-job-definition
• Job Queue
  • CloudFormation parameter "prefix" value-job-queue

Below is an example of the minimum necessary parameter settings for arm64. Overwrite the CMD [ "" ] in Dockerfile to set the options for the layer creation shell.

[ "-p s3://publish-lambda-layer-tool-pcjkn63zhk/python/requirements.txt", "-n http_request_lib_test_python_3_12_arm64", "-s lambda-layer-pcjkn63zhk" ]

Options for the Layer Creation Shell

For example, I prepared requirements.txt in s3://publish-lambda-layer-tool-pcjkn63zhk/python/. Its contents are as follows:

requests == 2.32.3
httpx == 0.27.2

You can also use the "-r" option without preparing a requirements.txt file by specifying it directly.

[ "-r requests == 2.32.3|httpx == 0.27.2", "-n http_request_lib_test_python_3_12_arm64", "-s lambda-layer-pcjkn63zhk" ]

Below is an example of the minimum necessary parameter settings for x86_64.

[ "-p s3://publish-lambda-layer-tool-pcjkn63zhk/python/requirements.txt", "-n http_request_lib_test_python_3_12_x86_64", "-s lambda-layer-pcjkn63zhk" ]

If the status becomes Succeeded, it is successful.

The layer zip file is stored in S3.

The metadata of the layer zip file in S3 contains various information about the layer.

It is also registered as a Lambda layer.

Create a function specifying the desired runtime and architecture.

Add the layer to the function.

The default timeout for Lambda is 3 seconds, so change it to about 1 minute just in case.

Try using the modules (requests, httpx) brought in by the layer. Additionally, use platform.python_version() to get the actual running Python version.

import platform
import httpx
import requests

def lambda_handler(event, context):
    python_version = platform.python_version()
    print('[lambda python runtime version] ' + python_version)
    url = 'https://aws.amazon.com/'
    httpx_response = httpx.get(url)
    print('[httpx response]')
    print(httpx_response)
    requests_response = requests.get(url)
    print('[requests response]')
    print(requests_response)
    return python_version

The HTTP request goes through, and the running Python version is returned, indicating success, so the layer is usable.

Since the base image for layer creation (3.12.7) was released shortly before, it seems that the actual Lambda (3.12.5) had not yet caught up. If it doesn't work well when the Python versions differ, try selecting a base image that matches the Python version and recreating the layer.

Although this was implemented in Python, I believe that by replacing the contents of the layer creation tool (only the Dockerfile and shell), layers for other Lambda-supported languages can also be easily created using this method. Additionally, the CloudFormation templates for the image creation environment CodeBuild and the container execution environment AWS Batch were created to be used generically, not specifically for the layer creation tool. Therefore, you can use them to freely prepare tools and applications in any language.