Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Packaging] Support Python 3.11 #26923

Merged
merged 31 commits into from
Oct 8, 2023
Merged

[Packaging] Support Python 3.11 #26923

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
edf18bd
Init
bebound Jul 18, 2023
e2852ec
Replace inspect.getargspec
bebound Jul 18, 2023
d0421af
Merge branch 'dev' into python-3.11
bebound Jul 20, 2023
c5dec5a
Fix mock_get_extension
bebound Jul 20, 2023
f91ee23
Fix mock error
bebound Jul 21, 2023
db4341f
Test new knack
bebound Jul 24, 2023
7a0c285
Fix test_parser_error_spellchecker
bebound Jul 24, 2023
265fb6c
Fix test_help_loads
bebound Jul 24, 2023
70a89dc
Fix test_command_index_always_loaded_extension
bebound Jul 25, 2023
f2dc5e7
Fix test_vm_defaults
bebound Jul 25, 2023
890df61
Revert "Test new knack"
bebound Jul 26, 2023
97beb14
Merge remote-tracking branch 'upstream/main' into python-3.11
bebound Aug 1, 2023
ae79bd7
Merge remote-tracking branch 'upstream/dev' into python-3.11
bebound Aug 3, 2023
9532240
Revert "Fix test_vm_defaults"
bebound Aug 3, 2023
5a3d9f2
Merge remote-tracking branch 'upstream/dev' into python-3.11
bebound Aug 3, 2023
957f19b
Use 3.11 in test
bebound Aug 4, 2023
0e92cb2
Fix enum str change in 3.11
bebound Aug 4, 2023
edea742
Merge remote-tracking branch 'upstream/dev' into python-3.11
bebound Aug 4, 2023
7e19a72
Minor fix
bebound Aug 7, 2023
36d8dc4
Use inspect.signature()
bebound Aug 7, 2023
87d3e90
Ignore deprecated-method
bebound Aug 7, 2023
952aac6
Merge remote-tracking branch 'upstream/dev' into python-3.11
bebound Aug 10, 2023
b5fedff
Merge branch 'dev' into python-3.11
bebound Aug 17, 2023
f4cf181
Merge remote-tracking branch 'upstream/dev' into python-3.11
bebound Aug 24, 2023
c8109ff
Rollback to inspect.getfullargspec
bebound Aug 25, 2023
ba4951a
Replace locale.getdefaultlocale with locale.getlocale
bebound Sep 12, 2023
f87722c
Merge commit 'a0b940589ec808e41b379b67286fe6d975ed621f' into python-3.11
bebound Sep 13, 2023
f2baaba
Use 3.10 in CodegenCoverage
bebound Sep 13, 2023
6936dc5
Revert fix mock_get_extension
bebound Sep 13, 2023
8b7a8c3
Use 3.11 in CodegenCoverage
bebound Sep 14, 2023
d0887b9
Merge remote-tracking branch 'upstream/dev' into python-3.11
bebound Sep 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions azure-pipelines-full-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- template: .azure-pipelines/templates/automation_test.yml
parameters:
Expand All @@ -44,8 +44,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- template: .azure-pipelines/templates/automation_test.yml
parameters:
Expand All @@ -63,8 +63,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- template: .azure-pipelines/templates/automation_test.yml
parameters:
Expand Down Expand Up @@ -107,8 +107,8 @@ jobs:
fullTest: true
jobName: 'FullTest'

- job: AutomationFullTestPython310ProfileLatest
displayName: Automation Full Test Python310 Profile Latest
- job: AutomationFullTestPython311ProfileLatest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should drop Python 3.10 tests right away as the bundled Python is still 3.10.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bundled Python is also bumped in #26749

displayName: Automation Full Test Python311 Profile Latest
timeoutInMinutes: 9999
strategy:
maxParallel: 8
Expand Down Expand Up @@ -147,7 +147,7 @@ jobs:
- AutomationTest20190301
- AutomationTest20180301
- AutomationFullTestPython39ProfileLatest
- AutomationFullTestPython310ProfileLatest
- AutomationFullTestPython311ProfileLatest
condition: and(failed(), in(variables['Build.Reason'], 'BatchedCI'))
displayName: Notify CI Errors
pool:
Expand Down
20 changes: 10 additions & 10 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,8 +455,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- template: .azure-pipelines/templates/automation_test.yml
parameters:
Expand All @@ -472,8 +472,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- template: .azure-pipelines/templates/automation_test.yml
parameters:
Expand All @@ -492,8 +492,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- task: UsePythonVersion@0
displayName: 'Use Python $(python.version)'
Expand All @@ -513,8 +513,8 @@ jobs:
name: ${{ variables.ubuntu_pool }}
strategy:
matrix:
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
steps:
- task: UsePythonVersion@0
displayName: 'Use Python $(python.version)'
Expand Down Expand Up @@ -1015,8 +1015,8 @@ jobs:
matrix:
Python39:
python.version: '3.9'
Python310:
python.version: '3.10'
Python311:
python.version: '3.11'
pool:
name: ${{ variables.ubuntu_pool }}
steps:
Expand Down
1 change: 1 addition & 0 deletions scripts/ci/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ CLASSIFIERS = [
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
]

Expand Down
8 changes: 4 additions & 4 deletions src/azure-cli-core/azure/cli/core/tests/test_command_registration.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,10 +160,10 @@ def _mock_get_extension_modname(ext_name, ext_dir):
return "azext_always_loaded"

def _mock_get_extensions():
MockExtension = namedtuple('Extension', ['name', 'preview', 'experimental', 'path', 'get_metadata'])
return [MockExtension(name=__name__ + '.ExtCommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}),
MockExtension(name=__name__ + '.Ext2CommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}),
MockExtension(name=__name__ + '.ExtAlwaysLoadedCommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {})]
MockExtension = namedtuple('Extension', ['name', 'preview', 'experimental', 'path', 'get_metadata', 'version', 'ext_type'])
return [MockExtension(name=__name__ + '.ExtCommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}, version='0.0.1', ext_type='dev'),
MockExtension(name=__name__ + '.Ext2CommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}, version='0.0.1', ext_type='dev'),
MockExtension(name=__name__ + '.ExtAlwaysLoadedCommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}, version='0.0.1', ext_type='dev')]
Copy link
Contributor Author

@bebound bebound Jul 20, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MockExtension does not mock all attribution in Extension.

Fix stderr in "Unit Test for Core 3.11" when run azdev test azure-cli-core

ERROR cli.azure.cli.core:init.py:273 Error loading command module 'serviceconnector': 'Extension' object has no attribute 'version'

Error stack is:

Traceback (most recent call last):
  File "c:\users\kk\developer\azure-cli\src\azure-cli-core\azure\cli\core\__init__.py", line 256, in _update_command_table_from_modules
    module_command_table, module_group_table = _load_module_command_loader(self, args, mod)
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\kk\developer\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 1085, in _load_module_command_loader
    return _load_command_loader(loader, args, mod, 'azure.cli.command_modules.')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\kk\developer\azure-cli\src\azure-cli-core\azure\cli\core\commands\__init__.py", line 1052, in _load_command_loader
    module = import_module(prefix + name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1264.0_x64__qbz5n2kfra8p0\Lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "c:\users\kk\developer\azure-cli\src\azure-cli\azure\cli\command_modules\serviceconnector\__init__.py", line 7, in <module>
    from azure.cli.command_modules.serviceconnector._help import helps  # pylint: disable=unused-import
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\kk\developer\azure-cli\src\azure-cli\azure\cli\command_modules\serviceconnector\_help.py", line 70, in <module>
    if not should_load_source(source):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\kk\developer\azure-cli\src\azure-cli\azure\cli\command_modules\serviceconnector\_utils.py", line 34, in should_load_source
    installed_extensions = [item.get('name') for item in list_extensions()]
                                                         ^^^^^^^^^^^^^^^^^
  File "c:\users\kk\developer\azure-cli\src\azure-cli-core\azure\cli\core\extension\operations.py", line 397, in list_extensions
    return [{OUT_KEY_NAME: ext.name, OUT_KEY_VERSION: ext.version, OUT_KEY_TYPE: ext.ext_type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\kk\developer\azure-cli\src\azure-cli-core\azure\cli\core\extension\operations.py", line 397, in <listcomp>
    return [{OUT_KEY_NAME: ext.name, OUT_KEY_VERSION: ext.version, OUT_KEY_TYPE: ext.ext_type,
                                                      ^^^^^^^^^^^
AttributeError: 'Extension' object has no attribute 'version'

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why didn't this fail before?

Copy link
Contributor Author

@bebound bebound Sep 13, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a Captured log call, it does not affect the test result.
The 3.11 test fails so I notice this error.

I've reverted this change.
I don't know cause of this error, I'll create a new PR if I can figure it out.


def _mock_load_command_loader(loader, args, name, prefix):

Expand Down
6 changes: 3 additions & 3 deletions src/azure-cli-core/azure/cli/core/tests/test_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,9 @@ def _mock_extension_modname(ext_name, ext_dir):
return ext_name

def _mock_get_extensions(**kwargs):
MockExtension = namedtuple('Extension', ['name', 'preview', 'experimental', 'path', 'get_metadata'])
return [MockExtension(name=__name__ + '.ExtCommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}),
MockExtension(name=__name__ + '.Ext2CommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {})]
MockExtension = namedtuple('Extension', ['name', 'preview', 'experimental', 'path', 'get_metadata', 'version', 'ext_type'])
return [MockExtension(name=__name__ + '.ExtCommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}, version='0.0.1', ext_type='dev'),
MockExtension(name=__name__ + '.Ext2CommandsLoader', preview=False, experimental=False, path=None, get_metadata=lambda: {}, version='0.0.1', ext_type='dev')]

def _mock_load_command_loader(loader, args, name, prefix):
from enum import Enum
Expand Down
2 changes: 1 addition & 1 deletion src/azure-cli-core/azure/cli/core/util.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -661,7 +661,7 @@ def get_arg_list(op):
sig = inspect.signature(op)
return sig.parameters
except AttributeError:
sig = inspect.getargspec(op) # pylint: disable=deprecated-method
sig = inspect.getfullargspec(op)
return sig.args


Expand Down
1 change: 1 addition & 0 deletions src/azure-cli-core/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
]

Expand Down
1 change: 1 addition & 0 deletions src/azure-cli-telemetry/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
]

Expand Down
2 changes: 1 addition & 1 deletion src/azure-cli-testsdk/azure/cli/testsdk/scenario_tests/utilities.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ def trim_kwargs_from_test_function(fn, kwargs):
# the next function is the actual test function. the kwargs need to be trimmed so
# that parameters which are not required will not be passed to it.
if not is_preparer_func(fn):
args, _, kw, _ = inspect.getargspec(fn) # pylint: disable=deprecated-method
args, _, kw, *_ = inspect.getfullargspec(fn)
Copy link
Contributor Author

@bebound bebound Jul 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FAILED src\azure-cli\azure\cli\command_modules\vm\tests\latest\test_vm_commands.py::VMSSCreateBalancerOptionsTest::test_vmss_create_default_app_gateway - AttributeError: module 'inspect' has no attribute 'getargspec'

inspect.getargspec is deprecated in 3.11. Use getfullargspec instead.
It returns FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations) instead of ArgSpec(args, varargs, keywords, defaults)

Ref:
https://docs.python.org/3.10/library/inspect.html#inspect.getfullargspec

Copy link
Member

@jiasli jiasli Jul 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's time to drop inspect.getargspec and even inspect.getfullargspec. See microsoft/knack#275 (comment).

A previous attempt was made by #24111.

if kw is None:
args = set(args)
for key in [k for k in kwargs if k not in args]:
Expand Down
1 change: 1 addition & 0 deletions src/azure-cli-testsdk/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
]

Expand Down
2 changes: 1 addition & 1 deletion .../cli/command_modules/keyvault/vendored_sdks/azure_keyvault_t1/key_vault_authentication.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ def __init__(self, authorization_callback):
# for backwards compatibility we need to support callbacks which don't accept the scheme
def _auth_callback_compat(self, server, resource, scope, scheme):
return self._user_callback(server, resource, scope) \
if len(inspect.getargspec(self._user_callback).args) == 3 \
if len(inspect.getfullargspec(self._user_callback).args) == 3 \
else self._user_callback(server, resource, scope, scheme)

def __call__(self, request):
Expand Down
1 change: 1 addition & 0 deletions src/azure-cli/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'License :: OSI Approved :: MIT License',
]

Expand Down