sam build --use-container fails with Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)} when building for ARM, but not for x86 #302

glez-aws · 2021-10-07T14:19:18Z

Description:

Hi,

very excited about arm64 Lambda support, thank you!

I am trying to build a Python Lambda function for the arm64 architecture on an AWS Cloud9 instance with SAM CLI, version 1.33.0.

I have used:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

to make my environment arm64-ready and then:

sam build --use-container

to build.

However, I get the following error:

Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)}

My code uses aws-xray-sdk==2.8.0 in requirements.txt, which triggers installing the dependency: wrapt==1.13.1.

I am able to successfully build:

without aws-xray-sdk in requirements.txt and with Architectures: ["arm64"] in template.yaml, and
with aws-xray-sdkin requirements.txt but without Architectures: ["arm64"] in template.yaml.

This tells me that my arm64 build environment seems to be working and also that building Python dependencies inside a container works, too, but not the combination of both.

Am I missing something or is there a bug that specifically affects pip-based builds in arm64 containers?

Thanks,
Constantin

Steps to reproduce:

Set up a Cloud9 instance.
Install SAM CLI, version 1.33.0.
Set up a simple, Python 3.8 based serverless app that uses Architectures: ["arm64"] in template.yaml to indicate that the Lambda function should be built for an arm64 architecture, and aws-xray-sdk==2.8.0 in its requirements.txt to trigger the installation of wrapt by the pip builder.
Run docker run --rm --privileged multiarch/qemu-user-static --reset -p yesto enable arm64 building.
Run sam build --use-container to trigger the error.

Observed result:

2021-10-07 13:44:10,323 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-10-07 13:44:10,325 | Using config file: samconfig.toml, config environment: default
2021-10-07 13:44:10,325 | Expand command line arguments to:
2021-10-07 13:44:10,326 | --template_file=/home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/template.yaml --use_container --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-10-07 13:44:10,383 | 'build' command is called
2021-10-07 13:44:10,384 | Starting Build inside a container
2021-10-07 13:44:10,394 | No Parameters detected in the template
2021-10-07 13:44:10,428 | 2 stacks found in the template
2021-10-07 13:44:10,429 | No Parameters detected in the template
2021-10-07 13:44:10,459 | 2 resources found in the stack 
2021-10-07 13:44:10,459 | No Parameters detected in the template
2021-10-07 13:44:10,490 | Found Serverless function with name='UpdateSESRulesFunction' and CodeUri='update_ses_rules/'
2021-10-07 13:44:10,490 | --base-dir is not presented, adjusting uri update_ses_rules/ relative to /home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/template.yaml
2021-10-07 13:44:10,491 | No Parameters detected in the template
2021-10-07 13:44:10,522 | Instantiating build definitions
2021-10-07 13:44:10,527 | Same function build definition found, adding function (Previous: BuildDefinition(python3.8, /home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/update_ses_rules, Zip, arm64, , 50c146ef-27af-468a-98cb-6fb1374e37ca, {}, {}, []), Current: BuildDefinition(python3.8, /home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/update_ses_rules, Zip, arm64, , 76dbab65-4985-448f-8993-89a520fdf356, {}, {}, []), Function: Function(name='UpdateSESRulesFunction', functionname='UpdateSESRulesFunction', runtime='python3.8', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/update_ses_rules', environment=None, rolearn=None, layers=[], events={'SSMParameterChanges': {'Type': 'EventBridgeRule', 'Properties': {'EventBusName': 'default', 'Pattern': {'source': ['aws.ssm'], 'detail-type': ['Parameter Store Change'], 'detail': {'name': ['/EmailRedirectorV2/ses_rule_set_name', '/EmailRedirectorV2/email_rules'], 'operation': ['Create', 'Update', 'Delete']}}}}}, metadata=None, inlinecode=None, codesign_config_arn=None, architectures=['arm64'], stack_path=''))
2021-10-07 13:44:10,532 | Building codeuri: /home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/update_ses_rules runtime: python3.8 metadata: {} architecture: arm64 functions: ['UpdateSESRulesFunction']
2021-10-07 13:44:10,533 | Building to following folder /home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/.aws-sam/build/UpdateSESRulesFunction

Fetching public.ecr.aws/sam/build-python3.8:latest-arm64 Docker container image......
2021-10-07 13:44:11,680 | Mounting /home/ec2-user/environment/EmailRedirectorV2/email-redirector-v2/update_ses_rules as /tmp/samcli/source:ro,delegated inside runtime container
Using the request object from command line argument
Loading workflow module 'aws_lambda_builders.workflows'
Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {botocore==1.21.56(wheel), aws-xray-sdk==2.8.0(wheel), boto3==1.18.46(wheel), jmespath==0.10.0(wheel), wrapt==1.13.1(sdist), jsonpickle==2.0.0(wheel), s3transfer==0.5.0(wheel), future==0.18.2(sdist), urllib3==1.26.7(wheel), six==1.16.0(wheel), python-dateutil==2.8.2(wheel)}
initial compatible: {botocore==1.21.56(wheel), python-dateutil==2.8.2(wheel), aws-xray-sdk==2.8.0(wheel), boto3==1.18.46(wheel), jmespath==0.10.0(wheel), jsonpickle==2.0.0(wheel), urllib3==1.26.7(wheel), six==1.16.0(wheel), s3transfer==0.5.0(wheel)}
initial incompatible: {future==0.18.2(sdist), wrapt==1.13.1(sdist)}
Downloading missing wheels: {future==0.18.2(sdist), wrapt==1.13.1(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp38 --dest /tmp/samcli/scratch future==0.18.2
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp38 --dest /tmp/samcli/scratch wrapt==1.13.1
compatible wheels after second download pass: {botocore==1.21.56(wheel), aws-xray-sdk==2.8.0(wheel), boto3==1.18.46(wheel), jmespath==0.10.0(wheel), jsonpickle==2.0.0(wheel), s3transfer==0.5.0(wheel), urllib3==1.26.7(wheel), six==1.16.0(wheel), python-dateutil==2.8.2(wheel)}
Build missing wheels from sdists (C compiling True): {future==0.18.2(sdist), wrapt==1.13.1(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/future-0.18.2.tar.gz
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/wrapt-1.13.1.tar.gz
compatible after building wheels (no C compiling): {botocore==1.21.56(wheel), aws-xray-sdk==2.8.0(wheel), boto3==1.18.46(wheel), jmespath==0.10.0(wheel), jsonpickle==2.0.0(wheel), s3transfer==0.5.0(wheel), future==0.18.2(wheel), urllib3==1.26.7(wheel), six==1.16.0(wheel), python-dateutil==2.8.2(wheel)}
Build missing wheels from sdists (C compiling False): {wrapt==1.13.1(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/wrapt-1.13.1.tar.gz
compatible after building wheels (C compiling): {botocore==1.21.56(wheel), aws-xray-sdk==2.8.0(wheel), boto3==1.18.46(wheel), jmespath==0.10.0(wheel), jsonpickle==2.0.0(wheel), s3transfer==0.5.0(wheel), future==0.18.2(wheel), urllib3==1.26.7(wheel), six==1.16.0(wheel), python-dateutil==2.8.2(wheel)}
Final compatible: {botocore==1.21.56(wheel), aws-xray-sdk==2.8.0(wheel), boto3==1.18.46(wheel), jmespath==0.10.0(wheel), jsonpickle==2.0.0(wheel), s3transfer==0.5.0(wheel), future==0.18.2(wheel), urllib3==1.26.7(wheel), six==1.16.0(wheel), python-dateutil==2.8.2(wheel)}
Final incompatible: {wrapt==1.13.1(wheel)}
Final missing wheels: {wrapt==1.13.1(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 43, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {wrapt==1.13.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 278, in run
    action.execute()
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 49, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {wrapt==1.13.1(sdist)}
Builder workflow failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 43, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {wrapt==1.13.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 278, in run
    action.execute()
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 49, in execute
    raise ActionFailedError(str(ex))
aws_lambda_builders.actions.ActionFailedError: {wrapt==1.13.1(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/__main__.py", line 118, in main
    builder.build(
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/builder.py", line 132, in build
    return workflow.run()
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 96, in wrapper
    func(self, *args, **kwargs)
  File "/usr/local/opt/sam-cli/lib/python3.8/site-packages/aws_lambda_builders/workflow.py", line 285, in run
    raise WorkflowFailedError(workflow_name=self.NAME, action_name=action.NAME, reason=str(ex))
aws_lambda_builders.exceptions.WorkflowFailedError: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)}
2021-10-07 13:46:20,181 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)}"}}

Build Failed
2021-10-07 13:46:20,496 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'b325b471-7ea3-46b7-b53d-c16e99e2e7a1', 'installationId': '92f62cbe-06bf-4366-af3a-5e066baf852b', 'sessionId': '24ddbccb-5431-446a-aa94-d1b3a7aa126b', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.33.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 130170, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
2021-10-07 13:46:21,035 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)}

Expected result:

A successful build.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Linux 4.14.232-123.381.amzn1.x86_64
sam --version: SAM CLI, version 1.33.0
AWS region: eu-central-1

The text was updated successfully, but these errors were encountered:

hawflau · 2021-10-14T16:05:38Z

Thanks for reporting the issue. I managed to reproduce the same result in Cloud9.
Looking at the log, it seems like Lambda Builder tried to build wrapt from source as there is no compatible wheel on pip. But for some reason it failed. I tried to run the same pip command in AL2 on an t4g instance and it succeeded. I guess it's likely a Lambda Builder issue. More investigation is needed.

I'm transferring to this issue to Lambda Builder.

eldritchideen · 2022-01-25T21:28:26Z

What seems to be occurring is that Lambda Builder is building the wheel for wrapt

aws-lambda-builders/aws_lambda_builders/workflows/python_pip/packager.py

Line 687 in 8bd869d

def build_wheel(self, wheel, directory, compile_c=True):

The wheel that gets created on my t4g instance is named wrapt-1.13.3-cp39-cp39-linux_aarch64.whl.
It seems that the builder then looks at all the wheels to ensure they are compatible. This is where wrapt gets failed. The code only considers wheels that match manylinux are compatible.

aws-lambda-builders/aws_lambda_builders/workflows/python_pip/packager.py

Line 440 in 8bd869d

match = re.match("manylinux_([0-9]+)_([0-9]+)_" + arch, perennial_tag)

heitorlessa · 2022-01-27T11:01:21Z

@praneetap @awood45 - As this is blocking us (Lambda Powertools) to offer ARM support, would you consider a PR to fallback to linux wheel if a manylinux wheel isn't found?

In our case, AWS Lambda Powertools depends on AWS X-Ray SDK which depends on Wrapt (C). Wrapt is commonly used by Python developers wanting to enforce decorator correctness. Wrapt builds wheels for multiple platforms but deliberately does not publish a manylinux wheel as it wouldn't support legacy Linux systems

@eldritchideen repro: https://github.com/eldritchideen/arm-python-lambda-example

praneetap · 2022-01-28T00:00:02Z

@heitorlessa this contribution would be awesome. Happy to support you in any way I can, @jfuss would be the dev poc to help with any questions.

jfuss · 2022-01-28T18:55:16Z

I don't this this is about manylinux specifically. Looking through a couple PEPs, there is a platform tag that is added to the .whl that is generated. This is left of to disutils which eventually get the uname of the system in some cases (not clear what paths actually lead here since I didn't go that deep).

It appears we handle this case specifically for x86:

aws-lambda-builders/aws_lambda_builders/workflows/python_pip/packager.py

Line 180 in 8bd869d

"linux_x86_64",

but looks like it was a miss when arm was added.

@heitorlessa I think this is pretty straightforward and is likely just adding the linux_aarch64 to

aws-lambda-builders/aws_lambda_builders/workflows/python_pip/packager.py

Line 175 in 8bd869d

"manylinux2014_aarch64",

and having some basic tests to validate. Are you up for this?

heitorlessa · 2022-01-28T19:05:01Z

Sure! @eldritchideen as you did the leg work and should be credited, do you wanna make the PR and I can do the first review and/or help with tests?

If you aren't free, I can look into it next Friday as I'm moving houses atm.

eldritchideen · 2022-01-28T20:38:50Z

@heitorlessa I'll take a look at the change and get a PR up.

mndeveci · 2022-04-12T17:58:55Z

Thanks for working on this issue @eldritchideen .

I confirmed that it is working as expected with latest version of SAM CLI which contains up to date lambda builders. Resolving this issue.

hawflau transferred this issue from aws/aws-sam-cli Oct 14, 2021

hawflau added area/workflow/python_pip type/bug labels Oct 14, 2021

konz-rs mentioned this issue Jan 9, 2022

Lambda SAM build (ARM64 archi) - Python libs fails aws/aws-sam-cli#3525

Closed

eldritchideen mentioned this issue Jan 26, 2022

docs: Add support and build instructions and workflows for arm64 aws-powertools/powertools-lambda-python#979

Closed

aws-sam-cli-bot added the maintainer/need-followup label Jan 31, 2022

eldritchideen mentioned this issue Feb 1, 2022

Fix python ARM build issue #321

Merged

brysontyrrell mentioned this issue Mar 17, 2022

PythonPipBuilder:ResolveDependencies errors on arm64, Incorrect Layer build behaviors aws/aws-sam-cli#3747

Closed

mndeveci closed this as completed Apr 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sam build --use-container fails with Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)} when building for ARM, but not for x86 #302

sam build --use-container fails with Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)} when building for ARM, but not for x86 #302

glez-aws commented Oct 7, 2021

hawflau commented Oct 14, 2021

eldritchideen commented Jan 25, 2022 •

edited

Loading

heitorlessa commented Jan 27, 2022

praneetap commented Jan 28, 2022

jfuss commented Jan 28, 2022

heitorlessa commented Jan 28, 2022

eldritchideen commented Jan 28, 2022

mndeveci commented Apr 12, 2022

sam build --use-container fails with Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)} when building for ARM, but not for x86 #302

sam build --use-container fails with Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.1(sdist)} when building for ARM, but not for x86 #302

Comments

glez-aws commented Oct 7, 2021

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

hawflau commented Oct 14, 2021

eldritchideen commented Jan 25, 2022 • edited Loading

heitorlessa commented Jan 27, 2022

praneetap commented Jan 28, 2022

jfuss commented Jan 28, 2022

heitorlessa commented Jan 28, 2022

eldritchideen commented Jan 28, 2022

mndeveci commented Apr 12, 2022

eldritchideen commented Jan 25, 2022 •

edited

Loading