Pillow sample on macOS

[marcelotduarte]

To Reproduce
Build and run the pillow sample on macOS

Expected behavior
Build and run.

Desktop (please complete the following information):

• Platform information (e.g. Ubuntu Linux 16.04): macOS
• OS architecture (e.g. amd64): 64
• cx_Freeze version [e.g. 6.6]: 6.6 / 6.7dev
• Python version [e.g. 3.9]: 3.6-3.9

Additional context
Part of my log:

2021-04-24T15:53:38.8672750Z *** WARNING ***
2021-04-24T15:53:38.8673820Z Attempting to copy two files to 'build/exe.macosx-10.14-x86_64-3.8/liblzma.5.dylib'
2021-04-24T15:53:38.8675830Z source 1: '/Users/runner/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/PIL/.dylibs/liblzma.5.dylib' (real: '/Users/runner/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/PIL/.dylibs/liblzma.5.dylib')
2021-04-24T15:53:38.8677720Z source 2: '/usr/local/opt/xz/lib/liblzma.5.dylib' (real: '/usr/local/Cellar/xz/5.2.5/lib/liblzma.5.dylib')
2021-04-24T15:53:38.8678680Z (This may be caused by including modules in the zip file that rely on binary libraries with the same name.)
2021-04-24T15:53:38.8679370Z Using only source 1.
2021-04-24T15:53:38.9058480Z + popd
2021-04-24T15:53:38.9059980Z + echo 'Run pillow sample'
2021-04-24T15:53:38.9060930Z ~/work/freeze-test/freeze-test
2021-04-24T15:53:38.9061470Z + TEST_PRECMD=./
2021-04-24T15:53:38.9062290Z + '[' darwin19 == msys ']'
2021-04-24T15:53:38.9063490Z + BUILD_DIR=/Users/runner/work/freeze-test/freeze-test/cx_Freeze/samples/pillow/build/exe.macosx-10.14-x86_64-3.8
2021-04-24T15:53:38.9065380Z + pushd /Users/runner/work/freeze-test/freeze-test/cx_Freeze/samples/pillow/build/exe.macosx-10.14-x86_64-3.8
2021-04-24T15:53:38.9066180Z + count=0
2021-04-24T15:53:38.9066550Z Run pillow sample
2021-04-24T15:53:38.9067990Z ~/work/freeze-test/freeze-test/cx_Freeze/samples/pillow/build/exe.macosx-10.14-x86_64-3.8 ~/work/freeze-test/freeze-test
2021-04-24T15:53:38.9071230Z ++ python /Users/runner/work/freeze-test/freeze-test/ci/build-test-json.py pillow 0
2021-04-24T15:53:38.9496640Z + TEST_NAME=test_pillow
2021-04-24T15:53:38.9498500Z + '[' -z test_pillow ']'
2021-04-24T15:53:38.9499260Z + [[ test_pillow == gui:* ]]
2021-04-24T15:53:38.9499760Z + [[ test_pillow == svc:* ]]
2021-04-24T15:53:38.9500160Z + ./test_pillow
2021-04-24T15:53:39.6799760Z Traceback (most recent call last):
2021-04-24T15:53:39.6803420Z   File "/Users/runner/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/cx_Freeze/initscripts/__startup__.py", line 81, in run
2021-04-24T15:53:39.6805290Z     module.run()
2021-04-24T15:53:39.6807130Z   File "/Users/runner/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/cx_Freeze/initscripts/Console.py", line 36, in run
2021-04-24T15:53:39.6808410Z     exec(code, m.__dict__)
2021-04-24T15:53:39.6809390Z   File "test_pillow.py", line 4, in <module>
2021-04-24T15:53:39.6810480Z   File "<frozen zipimport>", line 259, in load_module
2021-04-24T15:53:39.6812330Z   File "/Users/runner/hostedtoolcache/Python/3.8.9/x64/lib/python3.8/site-packages/PIL/Image.py", line 109, in <module>
2021-04-24T15:53:39.6813560Z     from . import _imaging as core
2021-04-24T15:53:39.6815920Z ImportError: dlopen(/Users/runner/work/freeze-test/freeze-test/cx_Freeze/samples/pillow/build/exe.macosx-10.14-x86_64-3.8/lib/PIL._imaging.cpython-38-darwin.so, 2): Library not loaded: @loader_path/.dylibs/libopenjp2.2.4.0.dylib
2021-04-24T15:53:39.6818750Z   Referenced from: /Users/runner/work/freeze-test/freeze-test/cx_Freeze/samples/pillow/build/exe.macosx-10.14-x86_64-3.8/lib/PIL._imaging.cpython-38-darwin.so
2021-04-24T15:53:39.6820280Z   Reason: image not found
2021-04-24T15:53:39.6873860Z ##[error]Process completed with exit code 255.

@cainesi You can test in your enviroment and maybe find a solution?

[cainesi]

The problem is that you're trying to run the result of the build_exe command. On OSX, you cx_freeze generally needs to update the dynamic links in the object files to make the frozen application run, but that does not happen until the bdist_mac command creates an application bundle.

The contents of the directory created by build_exe will often end up working anyways, since the dynamic links will still be pointing to the relevant files on the file system, but that is sort of an accident (and depends on the links). In your case, PIL._imaging.cpython-38-darwin.so was trying to load "@loader_path/.dylibs/libopenjp2.2.4.0.dylib". "@loader_path" is the directory of the linking object file. Since PIL._imaging.cpython-38-darwin.so is now in exe.macosx-10.14-x86_64-3.8/lib, rather than its original location, the link broke.

On my computer I get the same error when I try this, but the application bundle created by bdist_mac does work.

(This issue is unrelated to the WARNING at the top of the log.)

[marcelotduarte]

In my tests, the following samples are tested in osx:
["bcrypt", "cryptography", "icon", "matplotlib", "pandas", "pillow", "pycountry", "pydantic", "PyQt5", "pyside2", "pytz", "pyzmq", "simple", "sqlite", "tz"]
The GUI app I do have the screen, so I cannot confirm if it runs perfectly, but I have no errors.

Run PyQt5 sample
+ TEST_NAME=PyQt5app
~/work/freeze-test/freeze-test/cx_Freeze/samples/PyQt5/build/exe.macosx-10.14-x86_64-3.6 ~/work/freeze-test/freeze-test
+ '[' -z /Users/runner/work/freeze-test/freeze-test ']'
+ '[' PyQt5 == simple ']'
+ count=1
~/work/freeze-test/freeze-test
+ ./PyQt5app
++ python /Users/runner/work/freeze-test/freeze-test/ci/build-test-json.py PyQt5 1
+ TEST_NAME=
+ '[' -z '' ']'
+ popd
Unable to create basic Accelerated OpenGL renderer.
Unable to create basic Accelerated OpenGL renderer.
Core Image is now using the software OpenGL renderer. This will be slow.

The pillow sample is the only one that fails.

but that does not happen until the bdist_mac command creates an application bundle.

I do not agree with this. Maybe it's like that today, but I don't think it was like that originally. For example, on windows, build_exe works in all cases, it can run, and if you want you can use bdist_msi. You can use another app like innosetup to distribute, using the build. I think on mac it should be the same, bdist_mac is an option, not a must.

[cainesi]

I do not agree with this. Maybe it's like that today, but I don't think it was like that originally.

At least for the OSX, it looks like it has been that way since the very first version of macdist.py, about 10 years ago. The key thing is that setRelativeReferencePaths() is part of bdist_mac. I agree that there is no reason that we could not move that code over to the build_exe command (but only triggering on OSX), so that the relative references are fixed up at that point. The other platforms do not have the same issue as far as I know.

The situation is confusing for users, since the executable in the exe/ directory will often still work, because the dynamic links happen to still work after the files are moved into exe/ (though it is hit or miss, and the links might not be pointing to the "correct" copy of the libraries included in that directory). And sometimes the exe/ directory will work even when the .app bundle doesn't (e.g., because we're failing to copy something into the .app bundle).

[marcelotduarte]

The situation is confusing for users

It is confusing to me ;-)
I see now that I failed to help some users because I did not understand this.

[marcelotduarte]

I talked to @anthony-tuininga who said:
Regarding the macdist side of things -- my understanding of macOS is limited. In general, though, I agree that bdist_exe should work. Whether it uses portions of bdist_mac to generate an executable that can work is just fine -- but bdist_mac should not be the only method that works. You should be able to use build_exe and copy executables without having to package them.

[cainesi]

Hi Marcelo. No problem. I think it should be relatively easy to change the code so that the build_exe output runs correctly. I'll put that on my todo list for when I have a chance to turn back to this.

[marcelotduarte]

For pip installations, #1942 #1952
For conda #1942, #1946, #1951 and #1952