Use in AWS Lambda with minimal layer sizes

[dazza-codes]

It would help if the test-CI matrix and the setup dependencies supported the current versions of AWS Lambda for boto3 and botocore. It would be useful if one cell in a test-CI matrix pinned the boto3/botocore versions to the exact patch versions used in the current AWS Lambda runtime versions (and possibly ran in a container using the lambci runtime used by AWS Lambda for each python version in the runtimes available; excluding anything that does not support asyncio, of course). At the time of writing, the setup.py does not allow a range of botocore or boto3 versions that are compatible with the current AWS Lambda versions; the current AWS Lambda runtime versions are listed in [1] as follows.

[1] https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

Python runtimes
--

Python 3.8 | python3.8 | boto3-1.14.40 botocore-1.17.40 | Amazon Linux 2
Python 3.7 | python3.7 | boto3-1.14.40 botocore-1.17.40 | Amazon Linux
Python 3.6 | python3.6 | boto3-1.14.40 botocore-1.17.40 | Amazon Linux
Python 2.7 | python2.7 | boto3-1.14.40 botocore-1.17.40 | Amazon Linux

The setup.py contains the following at the time of writing:

install_requires = [
    # pegged to also match items in `extras_require`
    'botocore>=1.17.44,<1.17.45',
    'aiohttp>=3.3.1',
    'wrapt>=1.10.10',
    'aioitertools>=0.5.1',
]
extras_require = {
    'awscli': ['awscli==1.18.121'],
    'boto3': ['boto3==1.14.44'],
}

It could help if setup.py could keep track of AWS Lambda runtime versions and allow a range of versions that are compatible, e.g. maybe (?)

    'botocore>=1.17.40,<1.17.45',

    'boto3': ['boto3>=1.14.40,<1.14.45'],

Of course, doing so would entail that a test-CI matrix uses the actual AWS Lambda pinned versions to check that everything works OK with those releases. Then it should be possible to easily and confidently use aiobotocore in AWS Lambda layers and functions.

It might be interesting to try lambCI

• https://github.com/lambci/lambci
• https://stackshare.io/stackups/lambci-vs-travis-ci
• but it seems to require an AWS account for a CloudFormation deployment (it's not a service)

Motivation for Specific AWS Lambda Compatibility

Of course, it is always possible to bundle a different boto3/botocore into an AWS Lambda layer or function and carry on regardless of whatever the AWS Lambda runtime provides out of the box. However, sometimes a project may have so many dependencies or multiple layers that it exceeds the AWS package size limits [2]. In some serverless applications, the size of a lambda layer can approach the limits of layer sizes. In these cases, it helps to delete boto3 and all of its dependencies from a layer zip archive, because the lambda runtime container already provides them out-of-the-box. Doing so can save a big chunk of space in an AWS Lambda layer. The following example is the dep-tree for the current AWS Lambda boto3 version:

$ pipdeptree -p boto3
boto3==1.14.40
  - botocore [required: >=1.17.40,<1.18.0, installed: 1.17.40]
    - docutils [required: >=0.10,<0.16, installed: 0.15.2]
    - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.5]
    - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
      - six [required: >=1.5, installed: 1.14.0]
    - urllib3 [required: >=1.20,<1.26, installed: 1.25.9]
  - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.5]
  - s3transfer [required: >=0.3.0,<0.4.0, installed: 0.3.3]
    - botocore [required: >=1.12.36,<2.0a.0, installed: 1.17.40]
      - docutils [required: >=0.10,<0.16, installed: 0.15.2]
      - jmespath [required: >=0.7.1,<1.0.0, installed: 0.9.5]
      - python-dateutil [required: >=2.1,<3.0.0, installed: 2.8.1]
        - six [required: >=1.5, installed: 1.14.0]
      - urllib3 [required: >=1.20,<1.26, installed: 1.25.9]

[2] https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

[dazza-codes]

The docker images for lambci contain versions of boto3 that are not currently consistent with the published documentation.

published:

boto3==1.14.40
botocore==1.17.40

lambci container:

boto3==1.14.48
botocore==1.17.48

The setup.py could allow some more wiggle-room:

    'botocore>=1.17.40,<1.17.50'

    'boto3': ['boto3>=1.14.40,<1.14.50']

To check it ...

Using this lambda handler:

# lambda_versions.py
"""
Lambda Library Versions
***********************

This lambda function handler prints built-in package versions.

"""

import boto3
import botocore
import docutils
import jmespath
import dateutil
import s3transfer
import six
import urllib3

import os
import pprint
import sysconfig


def lambda_handler(event, context):

    os.system("ls -1d /var/runtime/*.dist-info | sort")
    print()

    sys_paths = sysconfig.get_paths()
    pprint.pprint(sys_paths)
    print()

    versions = {
        "boto3": boto3.__version__,
        "botocore": botocore.__version__,
        "docutils": docutils.__version__,
        "jmespath": jmespath.__version__,
        "dateutil": dateutil.__version__,
        "s3transfer": s3transfer.__version__,
        "six": six.__version__,
        "urllib3": urllib3.__version__,
    }
    pprint.pprint(versions)
    print()

    # pip requirements format
    for k, v in versions.items():
        print(f"{k}=={v}")
    print()

    return {"statusCode": 200, "body": "lambda versions done"}

WIth a current python3.6 lambci container:

mkdir -p /tmp/lambda_test
cp lambda_versions.py /tmp/lambda_test/
docker run --rm \
		-v /tmp/lambda_test/:/var/task:ro,delegated \
		lambci/lambda:python3.6 \
		lambda_versions.lambda_handler

It reports the latest container built-in library versions (although these may not be what is actually running on AWS Lambda at the time of writing):

/var/runtime/boto3-1.14.48.dist-info
/var/runtime/botocore-1.17.48.dist-info
/var/runtime/certifi-2020.6.20.dist-info
/var/runtime/chardet-3.0.4.dist-info
/var/runtime/docutils-0.15.2.dist-info
/var/runtime/idna-2.10.dist-info
/var/runtime/jmespath-0.10.0.dist-info
/var/runtime/python_dateutil-2.8.1.dist-info
/var/runtime/s3transfer-0.3.3.dist-info
/var/runtime/six-1.15.0.dist-info
/var/runtime/urllib3-1.25.10.dist-info

{'data': '/var/lang',
 'include': '/var/lang/include/python3.6m',
 'platinclude': '/var/lang/include/python3.6m',
 'platlib': '/var/lang/lib/python3.6/site-packages',
 'platstdlib': '/var/lang/lib/python3.6',
 'purelib': '/var/lang/lib/python3.6/site-packages',
 'scripts': '/var/lang/bin',
 'stdlib': '/var/lang/lib/python3.6'}

{'boto3': '1.14.48',
 'botocore': '1.17.48',
 'dateutil': '2.8.1',
 'docutils': '0.15.2',
 'jmespath': '0.10.0',
 's3transfer': '0.3.3',
 'six': '1.15.0',
 'urllib3': '1.25.10'}

boto3==1.14.48
botocore==1.17.48
docutils==0.15.2
jmespath==0.10.0
dateutil==2.8.1
s3transfer==0.3.3
six==1.15.0
urllib3==1.25.10

[thehesiod]

btw lambci is used under the hood in moto, I actually did a lot of work ensuring that would work

[github-actions]

This issue has been marked as stale because it has been inactive for more than 60 days. Please update this pull request or it will be automatically closed in 7 days.