Update setup script to be compatible with Py2exe for Python 3.7

[LeonarddeR]

Follow up of #7105 (comment)

Background

For years, NVDA has used Py2exe to package Python code into something that is executable on a system that doesn't have Python installed. For Python 2.7, we have been at Py2exe 0.6.9 (introduced in 2008) for a long time. For Python 3, a new version has been introduced in 2014 which looks like a rewrite. The last official release (0.9.2.2) was introduced in 2014.

Starting with Python 3.6 (end 2016) however, Python contains a backwards incompatible change of the byte code format that is not supported by the official py2exe.

Note that many parts of the build process of NVDA are handled by SCons. This includes:

• Compiling, linking and code signing all the additional libraries, such as NVDAHelper, liblouis, minhook, ESPeak
• Compiling ESpeak dictionaries, CLDR data and gettext translations
• Converting documentation to HTML using TXT2Tags, see Python 3: find a suitable and compatible replacement for txt2tags #8734

In summary, Scons deals with everything that is required to run NVDA from source, Py2exe is not involved in that process. Py2exe gets called as soon as a distribution of NVDA is created. The portable distribution that Py2exe creates is then packed into a launcher using NSiS.

Requirements

Before deciding on what packaging tool to use, this section lists the requirements for a packaging tool, based on the work that is now performed by Py2exe.

1. Byte compile every source file into a *.pyc or *.pyo file. Note that for Python 3, only the *pyc files remain.

2. Byte compile every Python dependency (comtypes, pyserial, etc.) required to run NVDA.

3. Collect Python extension modules (*.pyd files)

4. Bundle the several byte compiled files into a library.zip file in the main folder of the distribution.

5. Exclude several unneeded modules

6. Create executables from certain source files:

   source	destination(s)	description
   nvda.pyw	nvda_noUIAccess.exe, nvda_uiAccess.exe	Main executables (see below)
   nvda_eoaProxy.pyw	nvda_eoaProxy.exe	Used to integrate in Windows 7 EOA center
   nvda_slave.pyw	nvda_slave.exe	Used for add-on installation from shell, elevated rights during installation, etc.

7. Bundle a manifest with the executables. Theoretically, this can also be done using SCons, though in contrast to what I thought earlier, there is no code within the current scons structure of NVDA that can be reused for this.

8. Make sure that the UIAccess flag is added to the manifest of nvda_uiAccess.exe and nvda_eoaProxy.exe. Strictly spoken, this is code that has been added to our subclass of build_exe.py2exe in order to inject the UIAccess flag into the manifest created by py2exe. However, droppign py2exe in favour of another tool might force us to completely rewrite this logic.

9. Embed the NVDA logo into the executables as icon.

10. Set several version info flags in the version info resource of the executables.

11. Collect several system dll's not available on every system. This includes visual C++ redistributables.

12. Collect data files created using SCons, such as several libraries, language files, documentation, etc.

Alternatives

According to research and ideas from others, there are four packaging/freezing options that can be used with Python 3. The following sections provide short descriptions along with pros and cons.

Py2exe fork by @albertosottile

See https://github.com/albertosottile/py2exe

Thankfully, @albertosottile contacted us in #8375 (comment), commenting that he picked up Py2exe where it has been abandoned in 2014. This resulted into a release of Py2exe 0.93.0. This version is said to work with Python 3.6 and 3.7.

Pros

• Stick to the same tool as used before. Though heavily changed under the hood, it still promises to be mostly compatible with older setup files. According to the original readme:

  It is planned to achive full compatibility with the setup-scripts for Python 2; however this is probably not yet the case.

• Probably the least efford if @albertosottile's fork proves to be working.

Cons

• A look at the code of Py2exe for Python 3 seems to reveal that the support to create manifests is not yet implemented, that is, there is lots of commented out code that seems to origin from the Python 2 version. Having said that, the logic to embed resource files in the executable still seems to be available and an example setup files shows an alternative method to embed the manifest. It's just not compatible with the current approach in our setup.py.
• @albertosottile's fork of this project is pretty new. Some details:
  • It is not yet listed on PyPI. Note the version number
  • It has only 3 watchers, 5 stars and 2 forks on github
  • It has an open issue regarding wxPython support.

Requirements scheme

Feature	supported
Create multiple executables	Yes
Bundle manifest	Yes, but differently than before, might require pr to upstream
UIAccess	No explicit support, see manifest
Embed logo	Yes
Set version info	Yes
Collect system dll's	Yes
Collect external Python dependencies and *.pyd files	Yes
Collect data files	Yes
Library file	Yes

PyInstaller

PyInstaller is an alternative tool to build executables. It also supports byte code encryption, though that seems not very useful in the case of NVDA.

Pros

• This tool supports most functionality we need out of the box, including stuff that's now handled by scons, such as:
  • Code signing
  • Full automatic support for CRTs
  • Manifests with UAC
• It is very actively maintained.

Cons

• Building multiple executables at once seems to be broken

• It does not integrate very well with distutils. It looks like the PyInstaller documentation suggests using a batch file to write down command line commands. Alternatively, you can use a spec file. Writing a setup.py for your package seems unsupported. Having said that, we're now calling a python interpretter from scons to execute the setup script, so basically, we're using a command line call already. This is therefore no show stopper, just a change of approach.

• More importantly, I've seen ridiculous tracebacks with one of my test applications referring to pyinstaller:

   Traceback (most recent call last):
     File "main.py", line 1, in <module>
     File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in 		exec_module
       exec(bytecode, module.__dict__)
     File "core.py", line 5, in <module>
     File "c:\python36\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 631, in exec_module
       exec(bytecode, module.__dict__)
  

  It looks like every import is somehow redirected to pyInstaller. I'm pretty sure this will cause issues with our use of comtypes com interface modules, globalPlugins, etc. which rely on appending paths to the module path variable. Showing raw paths to the python directory also looks a bit cheap, but I recall that with py2exe, we also had to work around this.

Requirements scheme

Here is the official features page from PyInstaller

Feature	supported
Create multiple executables	No, broken in v3.0, see pyinstaller/pyinstaller#1527
Bundle manifest	Yes
UIAccess	Yes
Embed logo	Yes
Set version info	Yes
Collect system dll's	Yes
Collect external Python dependencies and *.pyd files	Yes
Collect data files	Yes
Library file	No, seems to require a custom implementation using scripting according to features section, but no information found in documentation so far

cx_Freeze

This is an alternative that integrates well with distutils. Note that it is far from compatible with py2exe, so setup.py would still require an overhaul. Running cx_Freeze also results into a package structure that is very similar to the one py2exe creates, including a librar .zip file in the lib sub directory. The default settings do not create a zip file, but that can be customized easily.

cx_Freeze is hosted on GitHub. We can include it as a separate submodule. Though it requires some c code to be compiled, that shouldn't be that much of a problem and takes less than 20 seconds, so that doesn't justify an inclusion in misc deps in my opinion.

Pros

• We're using it for a @BabbageCom Python project including wx and comtypes, and it works pretty well for our aims. So, I do have experience with it.
• It can replace paths shown in tracebacks
• Can include MSVC redistributable automatically
• Can include additional files in the library zip file

Cons

• No official build supporting Python 3.7, we'd have to build from source.

• There is no built-in support for manifests and setting the UI Access privilege level. We will probably have to implement this manually, either by expanding setup.py or by building the base executables ourselves.

• By default, cx_freeze assumes that all python stuff is in a lib sub directory (i.e. in lib/library.zip) rather than in the root of the folder where the executables live. This is not easily customisable and seems to be hard-coded behaviour. We're using the lib folder for our x86 libraries, storing the python library in there might be a bit ugly.

• When there is a startup error, cx_Freeze shows an error message box including the name of cx_Freeze itself. This might be confusing for users.

• The last commit for the project dates from september 2018. There are 260 open issues, including two from myself:

  • Support integrating a manifest into Windows executables marcelotduarte/cx_Freeze#385
  • Allow storing library.zip and other depedencies alongside the main application marcelotduarte/cx_Freeze#413

  Both issues haven't gotten any attention.

Requirements scheme

Feature	supported
Create multiple executables	Yes
Bundle manifest	No, should probably be embedded in base executables
UIAccess	No
Embed logo	Yes
Set version info	Yes
Collect system dll's	Yes
Collect external Python dependencies and *.pyd files	Yes
Collect data files	Yes
Library file	Yes

Nuitka

Suggestion by @ctoth in #8375 (comment)

Nuitka chooses a completely different approach. Instead of compiling the Python code to byte code and bundling a loader with it, it converts all code to c and then compiles it using VC++ and links it against the Python library. It also claims to be faster than CPython.

PROS

• Nuitka might improve performance.

Cons

• As all python code has to be converted to C++ and then compiled, creating a distribution of NVDA is going to be a time-consuming process. We might be able to provide pre-compiled obj files for static dependencies (such as wx and comtypes). This requires some investigation, to find out and is currently undocumented.
• It looks like Nuitka doesn't support changing the path of modules. I'm not even sure whether it would be possible to make Nuitka import python code bundled in add-ons. This is something to find out if we decide to explore this direction further.
• THere is a huge runtime difference between running NVDA from source and from a distribution.

Requirements scheme

Feature	supported
Create multiple executables	Yes
Bundle manifest	No
UIAccess	No
Embed logo	Yes
Set version info	No
Collect system dll's	No
Collect external Python dependencies and *.pyd files	Unsure
Collect data files	No
Library file	No

HO to proceed?

Based on the above outline, I propose the following order of investigation of our options:

1. Devote time to investigate using the Py2exe fork by @albertosottile. He seems to be pretty interested in our possible efforts to freeze NVDA with his fork. It might also be the least effort on our end. However, it would help to know what his goals are with the project, and whether he's willing to maintain it in the near and far future. @albertosottile: Would you be able to elaborate on this?
2. Try to get in touch with the PyInstaller developers to point them at the main cons listed above. The most major question would be whether there would be support for addons and custom comtypes com interfaces.
3. Try to get in touch with @anthony-tuininga to ask him about whether he's intending to continue the active development of cx_Freeze.
4. Based on the outcome of 2 and 3 when 1 fails, decide between PyInstaller and cx_Freeze. In the current state, cx_Freeze has my preference, but that preference may change.

Note that option 1, 2 and 3 could be worked on in parallel. I'm tempted to consider Nuitka being out of scope for now.

[ctoth]

Explore the possibility of using Nuitka. It continues to improve, now has python 3.7 support, and the author is extremely extremely interested in working with people who have issues.

…

On Aug 6, 2018, at 12:01 PM, Leonard de Ruijter ***@***.***> wrote: @LeonarddeR commented on 8 Jun 2018, 08:16 CEST: cx_Freeze Compared to PyInstaller and py2exe, there are some drawbacks: There is nou built-in support for manifests and settings the UI Access privilege level. We will probably have to implement this manually, either by expanding setup.py or by building the base executables ourselves. By default, cx_freeze assumes that all python stuff is in a lib sub directory (i.e. in lib/library.zip) rather than in the root of the folder where the executables live. This is not easily customisable and seems to be hard-coded behaviour. I guess all this can be solved by building our own base executables, adding the manifests to these. This integrates quite well within the scons environment and might even lead to less complex code as the current code in setup.py. There is currently no Python 3.7 support (see marcelotduarte/cx_Freeze#399) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[LeonarddeR]

@ctoth commented on 6 Aug 2018, 20:41 CEST:

Explore the possibility of using Nuitka.
It continues to improve, now has python 3.7 support, and the author is extremely extremely interested in working with people who have issues.

I've played a bit with Nuitka. It is a really impressive and interesting piece of software.

A major downside is that all python modules are converted to c++ code and than recompiled. Though a major performance boost is promised, it takes a significant time to run the distribution creation process. I also got errors like this:

fatal error C1002: compiler is out of heap space in pass 2

It might help when I start compiling again with less concurrent jobs. But the time increase is concerning, as it will take much more time to create try builds, etc.

[LeonarddeR]

Hmm, integrating the creation of a custom cx_freeze base in our scons workflow isn't as easy as I thought, it causes weird string conversion warnings I don't expect, probably because I'm missing some essential requirements to have scons build stuff that should be linked against the python library.

I filed marcelotduarte/cx_Freeze#413

[LeonarddeR]

I did some additional research into pyinstaller,, but at least its default configuration shows full paths in trace backs and raises disgusting PyInstallerImportErrors. Furthermore, it seems it is either a nvda.exe with everything included, or a nvda.exe with nothing included. Last but not least, the installation creation process doesn't feel very pythonic.

It seems it is either something that is ugly but has features (pyinstaller), or something that is much more elegant but has less features than py2exe and hasn't received official releases during last year (CX_freeze). The master branch of the latter is python 3.7 compatible, though.

[albertosottile]

If I may, I have been working on resurrecting py2exe and make it compatible with the latest versions of python. If you want, you could give a try at the prebuilt wheels that I just released here: https://github.com/albertosottile/py2exe/releases/tag/v0.9.3.0

Please, let me know if you find any errors when packaging NVDA. This is a big project and I am sure it will provide a good benchmark for these new wheels.

[LeonarddeR]

This is really interesting. We'll keep this in mind as soon as the Py3 transition begins.

[LeonarddeR]

For everyone who is interested in this discussion, I've updated #8375 (comment) to contain an in depth overview of our options to replace Py2exe 0.6.9.

@albertosottile: You seem to be pretty interested in our possible efforts to freeze NVDA with your fork. It might also be the least effort on our end. However, it would help to know what your goals are with the project, and whether you're willing to maintain it in the near and far future. Would you be able to elaborate on this?

[michaelDCurran]

I think a missing requirement is to collect any dependent python extention dlls (.pyd files). I don't think I saw that in your list?

[LeonarddeR]

I think a missing requirement is to collect any dependent python extention dlls (.pyd files). I don't think I saw that in your list?

You're correct, I'll at that.

[albertosottile]

You seem to be pretty interested in our possible efforts to freeze NVDA with your fork. It might also be the least effort on our end. However, it would help to know what your goals are with the project, and whether you're willing to maintain it in the near and far future. Would you be able to elaborate on this?

Thanks for the tag. I read the first post carefully and it seems that there are several potential issues in using the forked py2exe for your project, both technical and political. I will try to address them hereafter.

First of all, I never tried manifests and to be honest I was not even aware of this feature in py2exe. As far as I know, this could be the most serious technical hurdle you (we?) will probably have to face if you want to keep your existing script file. Alternatively, I am confident the same results could be achieved using resources, but that will need a rewrite of the script.

Second, I am not worried about the wxPython issue, since applying a workaround from your side is easy and, as I wrote here, it might not even be necessary in your case.

Thirdly, as you correctly noted, the fork is not listed on PyPI. This is also related to the small number of watchers/stars/forks. Honestly, I wanted the project to gain more traction before contacting the original maintainers and/or the PyPI team and get noticed. While it is true that the codebase seems to work now, I strongly believe that maintaining a project like py2exe is not a job for one single person. (Side note: look at what is happening to PyInstaller now, they barely have the time to fix the most urgent issues, let alone implementing new features). Hence, I do agree with your worries about the future maintenance of py2exe, not because I am not willing, but because I am totally sure that any effort from me alone will not be enough. Now, the way I see it, the more developers adopt this new py2exe, the more traction the project will gain. So, it would be a great deal if NVDA could use it, thus my interest in this migration. Nevertheless, I will completely understand if you would decide to go for an alternative with wider adoption.

Wrapping things up, I think a good way to proceed for you will be to experiment with the forked py2exe and see how far you can go using that, without applying excessive efforts on your existing codebase. In this way, if things work out, you will have a huge return for a quite small investment. Eventually, if it is possible to iron out all the issues without writing many fixes, you could even release a new version based on this py2exe, helping me gaining some traction, hence increasing the chances of its survival. Otherwise, you will still have time to look into the alternatives without much to regret for having tried.

[LeonarddeR]

Just a quick update. I've been able to create a distribution using the py2exe fork that started properly. There are many things to fix and improve on our end, but what I've been able to accomplish in under two hours is really impressive.

I will give a more thorough update halfway next week.

[LeonarddeR]

@feerrenrut: I'm getting the following error from py2exe when trying to integrate images/nvda.ico into the executables:

Exception

``` Traceback (most recent call last): File "setup.py", line 225, in + getRecursiveDataFiles('documentation', '../user_docs', excludes=('*.t2t', '*.t2tconf', '*/developerGuide.*')) File "c:\python37\lib\distutils\core.py", line 148, in setup dist.run_commands() File "c:\python37\lib\distutils\dist.py", line 966, in run_commands self.run_command(cmd) File "c:\python37\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "setup.py", line 97, in run super(py2exe, self).run() File "c:\python37\lib\site-packages\py2exe\distutils_buildexe.py", line 192, in run self._run() File "c:\python37\lib\site-packages\py2exe\distutils_buildexe.py", line 273, in _run builder.build() File "c:\python37\lib\site-packages\py2exe\runtime.py", line 235, in build self.build_exe(target, exe_path, options.libname) File "c:\python37\lib\site-packages\py2exe\runtime.py", line 377, in build_exe for res_type, res_name, res_data in BuildIcons(getattr(target, "icon_resources", ())): File "c:\python37\lib\site-packages\py2exe\icons.py", line 124, in BuildIcons grp_header = CreateGrpIconDirHeader(header, id_generator) File "c:\python37\lib\site-packages\py2exe\icons.py", line 90, in CreateGrpIconDirHeader raise ValueError("too many images for this icon: %d" % iconheader.idCount) ValueError: too many images for this icon: 11 ```

I'm very unfamiliar with the ico format. Is it true that there could be multiple images embedded in one ico? Before I continue investigation of this, it might help to know whether the ico file is really valid.

[LeonarddeR]

Reply to #8375 (comment) from @albertosottile

First of all, I never tried manifests and to be honest I was not even aware of this feature in py2exe. As far as I know, this could be the most serious technical hurdle you (we?) will probably have to face if you want to keep your existing script file. Alternatively, I am confident the same results could be achieved using resources, but that will need a rewrite of the script.

There is some commented out code in py2exe regarding manifests. I think I'll first try to have a look at that code in order to see whether it can be reused. If so, I can file a pr against your fork to have it included.

Second, I am not worried about the wxPython issue, since applying a workaround from your side is easy and, as I wrote here, it might not even be necessary in your case.

First investigations indeed seem to prove that we aren't going to suffer from this issue.

Thirdly, as you correctly noted, the fork is not listed on PyPI. This is also related to the small number of watchers/stars/forks. Honestly, I wanted the project to gain more traction before contacting the original maintainers and/or the PyPI team and get noticed. While it is true that the codebase seems to work now, I strongly believe that maintaining a project like py2exe is not a job for one single person. (Side note: look at what is happening to PyInstaller now, they barely have the time to fix the most urgent issues, let alone implementing new features). Hence, I do agree with your worries about the future maintenance of py2exe, not because I am not willing, but because I am totally sure that any effort from me alone will not be enough. Now, the way I see it, the more developers adopt this new py2exe, the more traction the project will gain. So, it would be a great deal if NVDA could use it, thus my interest in this migration. Nevertheless, I will completely understand if you would decide to go for an alternative with wider adoption.

I don't think that a lack of watchers and forks will hold us back from using your fork. IN fact, we've been using Py2exe 0.6.9 for over 10 years, and it is still pretty stable. Having said that, of course it would be good for us as well if your project gains more traction. May be NVDA can give this a boost somehow.

[albertosottile]

There is some commented out code in py2exe regarding manifests. I think I'll first try to have a look at that code in order to see whether it can be reused. If so, I can file a pr against your fork to have it included.

That's great news, and of course a PR would always be welcome.

[michaelDCurran]

Ico files contain multiple images suitable for various common display scenarios such as thumnails in File explorer, Desktop icon, Start menu icon, system tray icon etc. With newer Windows Operating Systems, the number of image sizes has gone up yet again. However, I can't completely guarantee that there is no maximum amount of images aloud in an ico file. To my knowledge our ico file is valid.

[LeonarddeR]

I couldn't found such a maximum for images in an ico file.

I've filed py2exe/py2exe#9.

There are two major things I still have to look into:

• The system dll routine has changed. System dll's like api-ms-win-crt-convert-l1-1-0.dll are ending up in the distribution
• The manifest inclusion logic has to be changed

[albertosottile]

I've filed albertosottile/py2exe#9.

PR accepted. Will be included in the next release.

There are two major things I still have to look into:

• The system dll routine has changed. System dll's like api-ms-win-crt-convert-l1-1-0.dll are ending up in the distribution

This occurs also with PyInstaller, so I assume the change was in the Python ABI, hence that is here to stay. The solution is to delete the api-*.dll files from the distribution after packaging. This can be done efficiently in the setup.py script, see e.g. here (L638).

[Adriani90]

@LeonarddeR should this issue be closed now that the referenced pull request has benn merged?

[feerrenrut]

Closed with #9605