Allow AWS Lambda botocore versions in latest releases

[dazza-codes]

https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

• currently provides boto3-1.16.31 and botocore-1.19.31

But the latest aiobotocore is tightly pinned to botocore>=1.20.49,<1.20.50, which is not compatible with the builtin AWS Lambda libraries. Why is aiobotocore so tightly pinned to one specific patch version of botocore? As a library, general practice is to allow it to be installed with a reasonable range of versions for dependencies. Could the setup.py file relax the version dependency on botocore to allow >=1.19.0?

If aiobotocore is going to be so tightly coupled to botocore, it should have a release schedule that is also tightly coupled to botocore so that every single release of botocore has a matching release for aiobotocore and the latter might as well use the same release version as botocore. This would allow packaging systems to find an aiobotocore that is compatible with any version of botocore.

The error motivating this issue is to minimize an AWS Lambda layer by using the AWS Lambda bundled libs for boto3/botocore and ensuring that the layer .zip file contains compatible versions of other libs with the bundled libs. But pip raises errors like:

The conflict is caused by:
    The user requested botocore~=1.19.0
    aiobotocore 1.3.0 depends on botocore<1.20.50 and >=1.20.49

[dazza-codes]

#834 was closed - IMO that's a mistake

[dusty-phillips]

I have a similar problem at the other end of the spectrum: I have a package that depends on a more recent version of botocore than 1.20.50.

Some insight into why the tight coupling exists and whether or not PRs that either expand the requirements or create a separate release for newer (or older, for that matter) versions would be welcome.

[ysaunier]

Same here, the restriction with botocore>=1.20.49,<1.20.50 it's really too restrictive (I'm not able to find a resolution with aws-sam-cli)

[TimSalmonAg]

Same. I have tried to use this package in two separate projects, but this package is unusable in both because it doesn't allow later patch versions like 1.20.78.

[dazza-codes]

I have some complex down-stream dependencies that cannot get resolved when trying to include aiobotocore and various other libraries that also depend on boto3 or botocore. I think my only option now is to fork this repo (ugh) and relax the constraints to publish a new library to pypi (ugh). I hate to have to do this, but I have other concerns to address and this might be the fastest way to get what I need. I'll probably regret it.

So, the adventure (or misadventure) begins with

• https://pypi.org/project/aio-botocore/ at aiobotocore 1.3.3
• published from https://github.com/dazza-codes/aiobotocore/tree/poetry-packaging

Hopefully this adventure succeeds, but it would be even better if it could die an early death with an upstream solution here.

The motivation for this is that poetry dependency resolution seems to never get solved in a project with a large number of dependencies. Using poetry update -vv shows that the problems are all in resolving versions of the boto3 and/or botocore libraries with the presence of aiobotocore that has really, really, really tight constraints. It can take literally 30 minutes and counting before I give up. Imagine how frustrating this is - so much that I've forked this repo and published a "mirror" library with relaxed constraints. Ugh.

This might be a pointless exercise, because a lot of our projects use dask with s3fs and:

s3fs==2021.8.0
  - aiobotocore [required: ~=1.4.0, installed: 1.4.0]
    - botocore [required: >=1.20.106,<1.20.107, installed: 1.20.106]

There's no escape 😱

PS. With s3fs < 0.5, it doesn't need aiobotocore.
PPS. By using aio-botocore, a large project with many dependencies and several that need botocore, apip install time came down from "forever" to < 1 min.

[thehesiod]

if you need s3fs + boto3/awscli, you should use their extras: https://github.com/dask/s3fs/blob/main/setup.py#L37

[ned2]

The tight pinning of botocore is definitely a barrier to people using any library that depends on aiobotocore. I find myself here because I wanted to use Pandas to write to S3, but this requires installing s3fs, which in turn depends on aiobotocore and I happen to have another dependency requiring botocore>=1.21.48.

So I'm blocked on using this functionality of Pandas in my current prioject.

[ned2]

Ah, so reading through this issue I see that these dependency constraints should be handled by pip's new resolver, but I'm using pip-tools, so it's still an issue until pip-tools gets the new pip resolver logic.

@dazza-codes and @thehesiod, I'm also trying to use s3fs. What I found worked was adding aiobotocore[boto3] to my requirements (and not adding the extra for s3fs)

[thehesiod]

the only way to provide some sort of guarantee is to tie aiobotocore to the version of botocore it was adapted to, there are too many internal changes that can happen we can't account for. If you want to take the risk feel free to do on your own but this library should not put you in the path of potential unforeseen breakages. The other thing we could do is vendor botocore however then this library would become enormous because we'd not only have to vendor botocore, but all its deps, and fix the namespacing of said libraries to not conflict with the user's copies. Best for now is ensuring we keep aiobotocore up to date which isn't too big a deal.

Here's a little bit of why the tight coupling: https://github.com/aio-libs/aiobotocore/blob/master/CONTRIBUTING.rst#background-and-implementation

best solution is for botocore to support async natively.

[vincentsarago]

for whom might read this issue.

The main issue with AWS Lambda is package size, and having two version of botocore is quite unworkable (because the package weight 60mb). Usually I remove both boto3 and botocore from the package to use the one provided by AWS Lambda env, but as explained in this issue, aiobotocore is strict about the version so it's pretty hard to make it work with AWS Lambda env.

Looking at the aiobotocore version, 2.1.2 require botocore>=1.23.24,<1.23.25 which is pretty close to botocore-1.23.32 present in AWS Lambda env, so I decided to use it in my AWS Lambda.

Here is the dockerfile I'm using to create the package.

ARG PYTHON_VERSION=3.9

FROM --platform=linux/amd64 public.ecr.aws/lambda/python:${PYTHON_VERSION}

WORKDIR /tmp

# Install dependencies
# HACK: aiobotocore has a tight botocore dependency
# https://github.com/aio-libs/aiobotocore/issues/862
# and because we HAVE TO remove both boto3 and botocore to save space for the package
# we have to force using old package version that seems `almost` compatible with Lambda env botocore
# https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

RUN pip install "pydantic" "rasterio" "rioxarray" "xarray" "zarr" "fsspec" "aiobotocore==2.1.2" "s3fs==2022.01.0" "boto3==1.20.24" "fsspec==2022.01.0" -t /asset --no-binary pydantic

# Reduce package size and remove useless files
RUN rm -rdf /asset/numpy/doc/ /asset/bin /asset/boto3* /asset/botocore*

COPY handler.py handler.py

CMD ["echo", "hello world"]

Note: This is a hack and you might experience edge effects, but I don't think the botocore API changed much between 1.23.24 and 1.23.32

[thehesiod]

@vincentsarago yep in fact it did not change. If you see later versions of botocore/awscli/boto3 that are api compatible (no new codepaths and test_patches.py does not error please submit PR to extend version ranges