Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task: Update to latest (most appropriate) version of Python #12064

Closed
feerrenrut opened this issue Feb 10, 2021 · 74 comments · Fixed by #15544
Closed

Task: Update to latest (most appropriate) version of Python #12064

feerrenrut opened this issue Feb 10, 2021 · 74 comments · Fixed by #15544
Assignees
Labels
api-breaking-change blocked/needs-external-fix blocked p2 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Milestone

Comments

@feerrenrut
Copy link
Contributor

feerrenrut commented Feb 10, 2021

  • Update appveyor.yml
  • Update to latest version of dependencies:
    • Scons
    • comtypes
    • wxPython
    • pySerial
    • py2exe
    • brltty

Initially Python 3.8 was being considered. 3.8 was chosen because 3.9 drops support for Windows 7 (see discussion: #10933 (comment)). The plan was to upgrade to windows 3.8, give Windows 7 users early notice, then upgrade again a year later.

Instead, we ran into a bug in python which prevented updating: (see https://bugs.python.org/issue38748 and python/cpython#26204)

When this is resolved (at the next compat breaking release) we will update to the latest (most appropriate version of Python).
If this is not fixed, we may have to consider moving NVDA to 64 bit, (the bug does not affect 64 bit python). This would be a large and risky change.

@feerrenrut
Copy link
Contributor Author

Fixed with PR Py3.8: Transparently use a Python virtual environment under the hood #12075

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Aug 26, 2021

This was reverted by #12298

@LeonarddeR LeonarddeR reopened this Aug 26, 2021
@LeonarddeR LeonarddeR added this to the 2022.1 milestone Aug 26, 2021
@zstanecic
Copy link
Contributor

zstanecic commented Aug 26, 2021 via email

@feerrenrut
Copy link
Contributor Author

Upgrading python in 2022.1 depends on the bug in libFFI being fixed, it isn't fixed yet, so I don't see much chance of this happening.

@LeonarddeR
Copy link
Collaborator

Upgrading python in 2022.1 depends on the bug in libFFI being fixed, it isn't fixed yet, so I don't see much chance of this happening.

Is there anything NV Access or the community could do to escalate this bug, or invest in fixing this ourselves?
I noticed that the EOL date of Python 3.7 will be in 2023, so that gives us at least one extra year. However, it looks like this is a lang standing bug now and there isn't much demand for others for this to be fixed any time soon.

@josephsl
Copy link
Collaborator

josephsl commented Aug 27, 2021 via email

@lukaszgo1
Copy link
Contributor

I noticed that the EOL date of Python 3.7 will be in 2023, so that gives us at least one extra year. However, it looks like this is a lang standing bug now and there isn't much demand for others for this to be fixed any time soon.

It is worth pointing out that new versions of Python 3.7 are no longer released as a binary copies therefore we are not keeping uptodate with them and are essentially stuck on 3.7.10 or whatever was the latest release provided as an installer. Has it been considered to compile Python ourselves to at least keep current with the security fixes?

@feerrenrut
Copy link
Contributor Author

Note, in addition to raising the bug we have created an automated test in cPython to reproduce the issue. We can ask cPython / libFFI developers to look at it. Otherwise, we may have to take this one our self.

@zstanecic
Copy link
Contributor

zstanecic commented Aug 27, 2021 via email

@lukaszgo1
Copy link
Contributor

I'm not sure I fully understood your last comment @zstanecic but in short what I've proposed above was to stay on Python 3.7 but use the latest releases by building them from sources since Python developers are no longer releasing binary versions of Py 3.7. Also it seems to me you're not realizing how serious the issue in libffi is - it affects much more than support for SAPI4 - there is no way to learn in advance in which cases NVDA would crash as a result of this bug.

@LeonarddeR
Copy link
Collaborator

According to https://bugs.python.org/issue38748 it looks like there's still a request open for a test to be provided using a pr.

@michaelDCurran
Copy link
Member

michaelDCurran commented Oct 27, 2021 via email

@mzanm
Copy link
Contributor

mzanm commented Nov 22, 2021

Hello.
I think upgrading to python 3.8 in 2023 wouldn't make sense, because by then python 3.8 and even 3.9 would be considered old especially because they won't be getting any bug fixes, only security ones, so then we're back to the same issue.

@michaelDCurran
Copy link
Member

michaelDCurran commented Nov 22, 2021 via email

@zstanecic
Copy link
Contributor

zstanecic commented Nov 22, 2021 via email

@feerrenrut
Copy link
Contributor Author

@zstanecic We have added automated tests to python (see https://bugs.python.org/issue38748 and python/cpython#26204) which show the issue, the problem is still not fixed.

I'll update the title of this issue to clarify.

@feerrenrut feerrenrut changed the title Task: Update to Python 3.8 Task: Update to latest (most appropriate) version of Python Nov 23, 2021
@josephsl
Copy link
Collaborator

josephsl commented Sep 5, 2023

Hi,

September 2023 update: even though we have Python 3.11.x, I advise 3.12 (October 2023) as it is released after Windows 8.1's end of support and will allow us to update to binary releases until mid-2025 unless critical regressions with dependencies are found.

Thanks.

@LeonarddeR
Copy link
Collaborator

@josephsl Could you please clarify your point about Windows 8.1? Will Python 3.12 still support Windows 8.1 including Server 2012 R2? Server 2012 R2 will receive extended security updates until 13 okt 2026.

@josephsl
Copy link
Collaborator

josephsl commented Sep 5, 2023 via email

@LeonarddeR
Copy link
Collaborator

I'm not strictly against it, but if we drop Windows 8.1 support as well, we really should consider our strategy regarding bitness of NVDA. In other words, I think there are hardly any x86 Windows 10+ users out there, and given several of our core dependencies are dropping X86 support for their wheels, a decision comes into view on this matter.
@seanbudd Would you be able to share usage statistics about Windows 8.1 and X86-versions of Windows 10+?

@josephsl
Copy link
Collaborator

josephsl commented Sep 5, 2023

Hi,

According to docs, Python 3.12 supports Windows 8.1 and later. I expect 3.13 to be the first release to require Windows 10 unless Python core devs say otherwise. Given that NV Access is no longer working on Windows 8.x specific bug fixes, I think we might as well take this opportunity to move onto Windows 10 and later as part of Python upgrade.

Thanks.

@lukaszgo1
Copy link
Contributor

It would be really great to have a clear statement from NV Access regarding what versions of Windows they intent to support with the Python upgrade, and if the migration to 64-bit is even something they consider for 2024.1. I am also curious what is the point of #15167 - as far as I can tell it merely duplicates content, and what we need is not a community discussion, but a clear announcement from NV Access. cc @seanbudd @michaelDCurran

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Sep 5, 2023

I believe the standpoint has always been that NVDA only drops support for particular Windows releases when the used techniques force us to. That's the reason why Windows 7 is supported until now, even though it has been deprecated for a long time now. SO if Python 3.12 supports Windows 8.1, there is no reason for NVDA not to.

Re 64-bit NVDA, that will be a long journey. Quick research revealed that al ctypes calls involving pointer types would require explicit parameter annotations. ctypes seems to assume 4 bytes for a parameter by default, so throwing an 64 bit pointer at it will result in an OverflowError to be raised. It is probably way quicker to invest in custom wheel building when necessary for wxPython, for example.
While I don't want to scatter this issue wit X64 ramblings, I recall there was an issue dedicated to X64 switching but unfortunately I can't find it.

@seanbudd
Copy link
Member

seanbudd commented Sep 7, 2023

We'd like to update to python 3.11 as it is likely to be more stable/safer than 3.12 and we want to start work on updating python very soon. It should be easy enough to update to 3.12 in 2025.1 or later.

We don't plan on fully dropping win 8.1 support, as python and windows still support it, even though we are no longer making bug fixes.

Dropping 32bit support is not on the current roadmap as it is a massive rewrite.

given several of our core dependencies are dropping X86 support for their wheels, a decision comes into view on this matter.

Could you give some examples here? wxPython has fixed this for now at least: wxWidgets/Phoenix#2246 (comment)

@josephsl
Copy link
Collaborator

josephsl commented Sep 7, 2023 via email

@LeonarddeR
Copy link
Collaborator

LeonarddeR commented Sep 8, 2023

Could you give some examples here? wxPython has fixed this for now at least: wxWidgets/Phoenix#2246 (comment)

I have missed that, I'm sorry.
The only other dependency I know of is Py2exe. There are still X86 wheels, but they are untested. See py2exe/py2exe#157

@XLTechie
Copy link
Collaborator

XLTechie commented Sep 8, 2023 via email

@LeonarddeR
Copy link
Collaborator

I wasn't very happy with pyinstaller five years ago. See #8375 (comment)

@XLTechie
Copy link
Collaborator

XLTechie commented Sep 8, 2023 via email

@mzanm
Copy link
Contributor

mzanm commented Sep 8, 2023

What about Nuitka? It compiles to pyd and needs VS or another compiler though...

@LeonarddeR
Copy link
Collaborator

I don't think switching away from py2exe is a serious point of discussion.

@seanbudd seanbudd assigned seanbudd and unassigned michaelDCurran Sep 27, 2023
seanbudd added a commit that referenced this issue Sep 28, 2023
In preparation for #12064

Summary of the issue:
Certain types which are imported for type checking only are not correctly encapsulated by strings.
threading.currentThread() is deprecated in favour of threading.current_thread()
with Exception() as ex: syntax is no longer valid in 3.11
in wxPython 4.2.0, integers are expected for scaling sizes
in wxPython 4.2.0, using AppendColumn is preferred with our current syntax of adding width
Description of user facing changes
None

Description of development approach
Makes various backwards compatible fixes that become compatibility issues when upgrading to python 3.11
@seanbudd seanbudd mentioned this issue Sep 28, 2023
5 tasks
seanbudd added a commit that referenced this issue Oct 16, 2023
Closes #12064
Closes #12551
Closes #15577
Closes #15167

Summary of the issue:
Python needs to be updated to 3.11, as Python 3.7 is EOL.
Python pip dependencies need to be updated to match the python upgrade.
typing_extensions is no longer needed.

Description of user facing changes
Performance and security enhancements from dependency upgrades.

Description of development approach
Updates python in build scripts
Updates pip dependencies
Set Windows 8.1 (Blue) as minimum windows version
Update references to python and windows version in docs
nose is replaced by the unittest module with xmlrunner to generate XML test output
drop python optimization to level 0 from level 1. level 1 removes asserts.
@lukaszgo1
Copy link
Contributor

@LeonarddeR wrote:

Re 64-bit NVDA, that will be a long journey. Quick research revealed that al ctypes calls involving pointer types would require explicit parameter annotations. ctypes seems to assume 4 bytes for a parameter by default, so throwing an 64 bit pointer at it will result in an OverflowError to be raised. It is probably way quicker to invest in custom wheel building when necessary for wxPython, for example. While I don't want to scatter this issue wit X64 ramblings, I recall there was an issue dedicated to X64 switching but unfortunately I can't find it.

While I cannot find it either, it would be good to create one with your initial thoughts. Even though there are no current plans to switch to 64-bit, we would certainly have to do so at some point, and having list of documented problems way in advance should be pretty helpful.

@marlon-sousa
Copy link
Contributor

Agreed, such an issue would be apreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api-breaking-change blocked/needs-external-fix blocked p2 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority triaged Has been triaged, issue is waiting for implementation.
Projects
None yet
Development

Successfully merging a pull request may close this issue.