Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNM:Upgrade awscli #1703

Closed

Conversation

GomathiselviS
Copy link
Collaborator

@GomathiselviS GomathiselviS commented Aug 15, 2023

SUMMARY

pip install awscli (version 1.23.0) results in the following error

Collecting PyYAML<5.5,>=3.10 (from awscli->-r /home/zuul/.ansible/collections/ansible_collections/amazon/aws/test-requirements.txt (line 16))
2023-08-16 17:45:17.417621 | controller |   Downloading PyYAML-5.4.1.tar.gz (175 kB)
2023-08-16 17:45:17.427972 | controller |      �������������������������������������� 175.1/175.1 kB 27.0 MB/s eta 0:00:00
2023-08-16 17:45:17.580233 | controller |   Installing build dependencies: started
2023-08-16 17:46:40.288496 | controller |   Installing build dependencies: still running...
2023-08-16 17:47:49.116379 | controller |   Installing build dependencies: still running...
2023-08-16 17:48:17.865355 | controller |   Installing build dependencies: finished with status 'done'
2023-08-16 17:48:17.865408 | controller |   Getting requirements to build wheel: started
2023-08-16 17:48:18.065668 | controller |   Getting requirements to build wheel: finished with status 'error'
2023-08-16 17:48:18.079112 | controller |   error: subprocess-exited-with-error
2023-08-16 17:48:18.079128 | controller |
2023-08-16 17:48:18.079134 | controller |   × Getting requirements to build wheel did not run successfully.
2023-08-16 17:48:18.079139 | controller |   │ exit code: 1
2023-08-16 17:48:18.079144 | controller |   ╰─> [68 lines of output]
2023-08-16 17:48:18.079149 | controller |       /var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
2023-08-16 17:48:18.079161 | controller |       !!
2023-08-16 17:48:18.079166 | controller |
2023-08-16 17:48:18.079171 | controller |               ********************************************************************************
2023-08-16 17:48:18.079176 | controller |               The license_file parameter is deprecated, use license_files instead.
2023-08-16 17:48:18.079181 | controller |
2023-08-16 17:48:18.079192 | controller |               By 2023-Oct-30, you need to update your project and remove deprecated calls
2023-08-16 17:48:18.079197 | controller |               or your builds will no longer be supported.
2023-08-16 17:48:18.079202 | controller |
2023-08-16 17:48:18.079206 | controller |               See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
2023-08-16 17:48:18.079211 | controller |               ********************************************************************************
2023-08-16 17:48:18.079216 | controller |
2023-08-16 17:48:18.079220 | controller |       !!
2023-08-16 17:48:18.079228 | controller |         parsed = self.parsers.get(option_name, lambda x: x)(value)
2023-08-16 17:48:18.079234 | controller |       running egg_info
2023-08-16 17:48:18.079239 | controller |       writing lib3/PyYAML.egg-info/PKG-INFO
2023-08-16 17:48:18.079243 | controller |       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
2023-08-16 17:48:18.079248 | controller |       writing top-level names to lib3/PyYAML.egg-info/top_level.txt
2023-08-16 17:48:18.079252 | controller |       Traceback (most recent call last):
2023-08-16 17:48:18.079256 | controller |         File "/home/zuul/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
2023-08-16 17:48:18.079262 | controller |           main()
2023-08-16 17:48:18.079266 | controller |         File "/home/zuul/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
2023-08-16 17:48:18.079271 | controller |           json_out['return_val'] = hook(**hook_input['kwargs'])
2023-08-16 17:48:18.079276 | controller |                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-16 17:48:18.079280 | controller |         File "/home/zuul/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
2023-08-16 17:48:18.079285 | controller |           return hook(config_settings)
2023-08-16 17:48:18.079289 | controller |                  ^^^^^^^^^^^^^^^^^^^^^
2023-08-16 17:48:18.079294 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel
2023-08-16 17:48:18.079299 | controller |           return self._get_build_requires(config_settings, requirements=['wheel'])
2023-08-16 17:48:18.079304 | controller |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-16 17:48:18.079308 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires
2023-08-16 17:48:18.079313 | controller |           self.run_setup()
2023-08-16 17:48:18.079317 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
2023-08-16 17:48:18.079322 | controller |           exec(code, locals())
2023-08-16 17:48:18.079326 | controller |         File "<string>", line 271, in <module>
2023-08-16 17:48:18.079331 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
2023-08-16 17:48:18.079338 | controller |           return distutils.core.setup(**attrs)
2023-08-16 17:48:18.079342 | controller |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-16 17:48:18.079347 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
2023-08-16 17:48:18.079351 | controller |           return run_commands(dist)
2023-08-16 17:48:18.079356 | controller |                  ^^^^^^^^^^^^^^^^^^
2023-08-16 17:48:18.079360 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
2023-08-16 17:48:18.079365 | controller |           dist.run_commands()
2023-08-16 17:48:18.079369 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
2023-08-16 17:48:18.079374 | controller |           self.run_command(cmd)
2023-08-16 17:48:18.079378 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1233, in run_command
2023-08-16 17:48:18.079385 | controller |           super().run_command(command)
2023-08-16 17:48:18.079390 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
2023-08-16 17:48:18.079394 | controller |           cmd_obj.run()
2023-08-16 17:48:18.079399 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 319, in run
2023-08-16 17:48:18.079403 | controller |           self.find_sources()
2023-08-16 17:48:18.079412 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 327, in find_sources
2023-08-16 17:48:18.079417 | controller |           mm.run()
2023-08-16 17:48:18.079421 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 549, in run
2023-08-16 17:48:18.079426 | controller |           self.add_defaults()
2023-08-16 17:48:18.079430 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 587, in add_defaults
2023-08-16 17:48:18.079435 | controller |           sdist.add_defaults(self)
2023-08-16 17:48:18.079440 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 113, in add_defaults
2023-08-16 17:48:18.079445 | controller |           super().add_defaults()
2023-08-16 17:48:18.079450 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
2023-08-16 17:48:18.079454 | controller |           self._add_defaults_ext()
2023-08-16 17:48:18.079459 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
2023-08-16 17:48:18.079463 | controller |           self.filelist.extend(build_ext.get_source_files())
2023-08-16 17:48:18.079468 | controller |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-08-16 17:48:18.079472 | controller |         File "<string>", line 201, in get_source_files
2023-08-16 17:48:18.079477 | controller |         File "/var/tmp/ansible-test-pip/pip-build-env-2xfqd32t/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
2023-08-16 17:48:18.079481 | controller |           raise AttributeError(attr)
2023-08-16 17:48:18.079486 | controller |       AttributeError: cython_sources
2023-08-16 17:48:18.079490 | controller |       [end of output]
2023-08-16 17:48:18.079497 | controller |
2023-08-16 17:48:18.079501 | controller |   note: This error originates from a subprocess, and is likely not a problem with pip.
2023-08-16 17:48:18.081338 | controller | error: subprocess-exited-with-error
ISSUE TYPE
  • Bugfix Pull Request
  • Docs Pull Request
  • Feature Pull Request
  • New Module Pull Request
COMPONENT NAME
ADDITIONAL INFORMATION

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.
https://ansible.softwarefactory-project.io/zuul/buildset/71bb48288f97401ba12143a4d3a8e3c7

✔️ ansible-galaxy-importer SUCCESS in 4m 14s
✔️ build-ansible-collection SUCCESS in 13m 40s
✔️ ansible-test-splitter SUCCESS in 5m 10s
Skipped 44 jobs

@ansibullbot ansibullbot added bug This issue/PR relates to a bug community_review integration tests/integration needs_triage small_patch Hopefully easy to review tests tests labels Aug 15, 2023
@ansibullbot
Copy link

@ansibullbot ansibullbot added module module plugins plugin (any type) labels Aug 16, 2023
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/9b0e45e84a764e38a58c704b77a4d3a0

✔️ ansible-galaxy-importer SUCCESS in 4m 46s
✔️ build-ansible-collection SUCCESS in 15m 35s
✔️ ansible-test-splitter SUCCESS in 5m 26s
integration-amazon.aws-1 FAILURE in 14m 20s
Skipped 43 jobs

@ansibullbot ansibullbot removed the small_patch Hopefully easy to review label Aug 16, 2023
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/f0e07e752f4f4d7eaa47b4e064216d5c

✔️ ansible-galaxy-importer SUCCESS in 5m 22s
✔️ build-ansible-collection SUCCESS in 13m 45s
✔️ ansible-test-splitter SUCCESS in 5m 24s
integration-amazon.aws-1 RETRY_LIMIT in 6m 55s
Skipped 43 jobs

@softwarefactory-project-zuul
Copy link
Contributor

This change depends on a change that failed to merge.

Change ansible/ansible-zuul-jobs#1819 is needed.

@GomathiselviS
Copy link
Collaborator Author

recheck

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/c0aaa5d562cc4d2591f587f024451939

✔️ ansible-galaxy-importer SUCCESS in 5m 25s
✔️ build-ansible-collection SUCCESS in 14m 17s
✔️ ansible-test-splitter SUCCESS in 5m 36s
integration-amazon.aws-1 FAILURE in 11m 37s
Skipped 43 jobs

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/d2f3b83509de46109a72687e9acaebcf

✔️ ansible-galaxy-importer SUCCESS in 4m 12s
✔️ build-ansible-collection SUCCESS in 14m 12s
✔️ ansible-test-splitter SUCCESS in 5m 06s
integration-amazon.aws-1 FAILURE in 9m 22s
Skipped 43 jobs

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/d2c1daf3e5784bc1a87878ae718745a9

✔️ ansible-galaxy-importer SUCCESS in 4m 50s
✔️ build-ansible-collection SUCCESS in 14m 34s
✔️ ansible-test-splitter SUCCESS in 5m 25s
integration-amazon.aws-1 FAILURE in 14m 11s
Skipped 43 jobs

Copy link
Contributor

@tremble tremble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awscli versions are closely tied to botocore/boto3 versions we can't just bump this version without bumping botocore and boto3 See the comment in constraints.txt

@GomathiselviS
Copy link
Collaborator Author

awscli versions are closely tied to botocore/boto3 versions we can't just bump this version without bumping botocore and boto3 See the comment in constraints.txt

Understood. I am trying a few workarounds to fix the cpython issue we see in amazon.aws integration tests on py3.11. This is not the final fix.

@GomathiselviS GomathiselviS marked this pull request as draft August 16, 2023 17:20
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/27500c854dac4393b3433a02e9c9c846

✔️ ansible-galaxy-importer SUCCESS in 4m 09s
✔️ build-ansible-collection SUCCESS in 14m 25s
✔️ ansible-test-splitter SUCCESS in 4m 59s
integration-amazon.aws-1 FAILURE in 11m 51s
Skipped 43 jobs

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/33f334ba1b3042d0b942940572a45d20

✔️ ansible-galaxy-importer SUCCESS in 5m 30s
✔️ build-ansible-collection SUCCESS in 13m 33s
✔️ ansible-test-splitter SUCCESS in 5m 39s
integration-amazon.aws-1 FAILURE in 14m 03s
Skipped 43 jobs

@ansibullbot ansibullbot added the module_utils module_utils label Aug 16, 2023
@alinabuzachis
Copy link
Collaborator

alinabuzachis commented Aug 16, 2023

@GomathiselviS Have you tried to upgrade/downgrade pyyaml version? It seems it is an error happening in pyyaml 5.4 yaml/pyyaml#724

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.
https://ansible.softwarefactory-project.io/zuul/buildset/dccec41259134a5d9d6011624d30e2d0

✔️ ansible-galaxy-importer SUCCESS in 4m 45s
✔️ build-ansible-collection SUCCESS in 14m 19s
✔️ ansible-test-splitter SUCCESS in 5m 16s
integration-amazon.aws-1 FAILURE in 29m 32s
✔️ integration-amazon.aws-2 SUCCESS in 38m 30s
✔️ integration-amazon.aws-3 SUCCESS in 23m 42s
✔️ integration-amazon.aws-4 SUCCESS in 35m 32s
✔️ integration-amazon.aws-5 SUCCESS in 25m 27s
✔️ integration-amazon.aws-6 SUCCESS in 42m 16s
✔️ integration-amazon.aws-7 SUCCESS in 22m 19s
✔️ integration-amazon.aws-8 SUCCESS in 34m 09s
✔️ integration-amazon.aws-9 SUCCESS in 41m 16s
✔️ integration-amazon.aws-10 SUCCESS in 32m 41s
integration-amazon.aws-11 RETRY_LIMIT in 1m 33s
✔️ integration-amazon.aws-12 SUCCESS in 21m 35s
integration-amazon.aws-13 RETRY_LIMIT in 1m 35s
✔️ integration-amazon.aws-14 SUCCESS in 21m 33s
integration-amazon.aws-15 FAILURE in 27m 24s
✔️ integration-amazon.aws-16 SUCCESS in 26m 50s
✔️ integration-amazon.aws-17 SUCCESS in 45m 43s
✔️ integration-amazon.aws-18 SUCCESS in 35m 22s
✔️ integration-amazon.aws-19 SUCCESS in 35m 05s
✔️ integration-amazon.aws-20 SUCCESS in 34m 39s
integration-amazon.aws-21 FAILURE in 31m 11s
✔️ integration-amazon.aws-22 SUCCESS in 25m 01s
integration-community.aws-1 FAILURE in 8m 24s
integration-community.aws-2 FAILURE in 12m 24s
integration-community.aws-3 FAILURE in 12m 12s
integration-community.aws-4 FAILURE in 11m 10s
integration-community.aws-5 FAILURE in 11m 31s
integration-community.aws-6 RETRY_LIMIT in 1m 32s
integration-community.aws-7 FAILURE in 11m 41s
integration-community.aws-8 FAILURE in 7m 56s
integration-community.aws-9 FAILURE in 10m 22s
integration-community.aws-10 FAILURE in 10m 25s
integration-community.aws-11 FAILURE in 11m 31s
integration-community.aws-12 FAILURE in 11m 20s
integration-community.aws-13 FAILURE in 10m 52s
integration-community.aws-14 FAILURE in 8m 22s
integration-community.aws-15 FAILURE in 9m 54s
Skipped 7 jobs

@GomathiselviS
Copy link
Collaborator Author

GomathiselviS commented Aug 17, 2023

@GomathiselviS Have you tried to upgrade/downgrade pyyaml version? It seems it is an error happening in pyyaml 5.4 yaml/pyyaml#724

Hi @alinabuzachis @tremble @jillr @gravesm

The issue mentioned in the description occurs with the current version of awscli that we're using, which is 1.23.0. This version comes with pyyaml 5.4, and we're facing a failure during pip installation of awscli. Unfortunately, we cannot simply pin pyyaml to a lower version due to security issues. (https://www.cvedetails.com/cve/CVE-2020-14343/)

I've attempted to resolve this issue by using --no-build-isolation with awscli, pyyaml, and cython, but it hasn't fixed the cython dependency problem. After considering the possible solutions, I believe that upgrading awscli to version 1 or migrating to awscli v2 might be the best course of action.

I've run some tests with the following versions, and most of the tests have passed. However, I still need to investigate the failed tests.

  • botocore==1.31.4
  • awscli==1.29.4
  • boto3==1.28.3

What do you think? Would upgrading awscli be a good solution, or do you have any other ideas? Let me know!

@gravesm
Copy link
Member

gravesm commented Aug 22, 2023

I think this one can be close. CI should be working again, aside from the current problems with zuul

@ansibullbot ansibullbot added needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR and removed community_review labels Aug 22, 2023
abikouo pushed a commit to abikouo/amazon.aws that referenced this pull request Sep 18, 2023
Bump release_version for eks_nodegroup

SUMMARY
Couldn't bump the version in ansible-collections#1415 - bumping it here
ISSUE TYPE

Docs Pull Request

COMPONENT NAME
eks_nodegroup
ADDITIONAL INFORMATION
abikouo pushed a commit to abikouo/amazon.aws that referenced this pull request Sep 18, 2023
Bump release_version for eks_nodegroup

SUMMARY
Couldn't bump the version in ansible-collections#1415 - bumping it here
ISSUE TYPE

Docs Pull Request

COMPONENT NAME
eks_nodegroup
ADDITIONAL INFORMATION
@tremble
Copy link
Contributor

tremble commented Oct 20, 2023

Looks like this was solved another way. Closing for now (we also bumped botocore in #1763)

@tremble tremble closed this Oct 20, 2023
abikouo pushed a commit to abikouo/amazon.aws that referenced this pull request Oct 24, 2023
Bump release_version for eks_nodegroup

SUMMARY
Couldn't bump the version in ansible-collections#1415 - bumping it here
ISSUE TYPE

Docs Pull Request

COMPONENT NAME
eks_nodegroup
ADDITIONAL INFORMATION
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue/PR relates to a bug integration tests/integration module_utils module_utils module module needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR plugins plugin (any type) python3 tests tests traceback
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants