Windows: setup.py not searching path for libraries

[jadczak]

OS: Windows 11
Python: PyPy 7.3.9
Pillow: 9.1.1 / 9.2.0dev

When trying to build from source the setup script checks a handful of locations for zlib and libjpeg, but doesn't bother searching path folders on Windows, resulting in the error: The headers or library files could not be found for zlib

Here is the output from pypy -mpip install Pillow is attached: pip_install_output.txt

I was able to successfully install Pillow with the following hack to the setup.py file:

diff --git a/setup.py b/setup.py
index 2701aa1c..47a20d7e 100755
--- a/setup.py
+++ b/setup.py
@@ -605,6 +605,11 @@ class pil_build_ext(build_ext):
             _add_directory(library_dirs, "/lib")
 
         if sys.platform == "win32":
+            # Hack
+            windows_path_dirs = [path_dir for path_dir in os.environ["path"].split(";") if "python" not in path_dir.lower()]
+            for windows_path_dir in windows_path_dirs:
+                _add_directory(library_dirs, windows_path_dir)
+                _add_directory(include_dirs, windows_path_dir)
             # on Windows, look for the OpenJPEG libraries in the location that
             # the official installer puts them
             program_files = os.environ.get("ProgramFiles", "")

Here is the successful output from pypy setup.py install -v: modified_setup_install_output.txt

and the resulting pip list output after install.

Package    Version
---------- ----------
cffi       1.15.0
greenlet   0.4.13
hpy        0.0.3
Pillow     9.2.0.dev0
pip        22.1.2
readline   6.2.4.1
setuptools 58.1.0
wheel      0.37.1

Naively searching the path folders, like I did, is likely less than ideal and requires a user to added the lib and include directories for both zlib and libjpeg separately (my path below for reference).

C:\pypy-test\Scripts
C:\Program Files\WindowsApps\Microsoft.PowerShell_7.2.5.0_x64__8wekyb3d8bbwe
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\WINDOWS\System32\OpenSSH\
C:\Program Files\dotnet\
C:\Program Files\Git\cmd
C:\Users\jadcz\.cargo\bin
C:\Users\jadcz\AppData\Local\Microsoft\WindowsApps
C:\Users\jadcz\AppData\Local\Programs\Microsoft VS Code\bin
C:\Users\jadcz\applications\Winpty-0.4.3\x64
C:\Program Files\Neovim\bin
C:\Program Files\CMake\bin
C:\Program Files\zlib\lib
C:\Program Files\zlib\include
C:\libjpeg-turbo64\lib
C:\libjpeg-turbo64\include

[radarhere]

Hi. So you're just running setup.py directly. This is not actually our recommended method for building from source on Windows.

https://pillow.readthedocs.io/en/stable/installation.html#building-on-windows

If you wish to compile Pillow manually, you can use the build scripts in the winbuild directory used for CI testing and development. These scripts require Visual Studio 2017 or newer and NASM.

[nulano]

In addition to @radarhere's comment, if you really wish to use setup.py anyway, you can set the LIB and INCLUDE variables, e.g.

set LIB=%LIB%;%PATH%
set INCLUDE=%INCLUDE%;%PATH%
pypy setup.py install -v:

[jadczak]

I was originally trying to install through pip using pypy -m pip install Pillow and when that failed I haphazardly started mucking about with the setup.py script based on the traceback I saw (it seemed like pip erroring out in setup.py so I started there).

I don't wish to use the setup.py, I'm lazy and would like to use pip, but that didn't work with my environment for reasons beyond me.

[nulano]

By default, pip will look for prebuilt wheels on PyPI. There is a Pillow 9.1.1 wheel uploaded for PyPy3.8-7.3.x (Pillow-9.1.1-pp38-pypy38_pp73-win_amd64.whl): https://pypi.org/project/Pillow/9.1.1/#files

I'm assuming you are using PyPy3.9-7.3.9, which is incompatible with that wheel.
Therefore pip falls back to a source build using setup.py.

As @radarhere pointed out above, triggering source builds directly (either with setup.py or via pip) is not recommended on Windows. Instead, it is recommended to use the scripts in winbuild.

Your analysis of the build error (when not using the winbuild scripts) is:

When trying to build from source the setup script checks a handful of locations for zlib and libjpeg, but doesn't bother searching path folders on Windows, resulting in the error: The headers or library files could not be found for zlib

My comment above demonstrates that you can easily pass the PATH variable to setup.py without modifying it.

[jadczak]

I think the disconnect for me is that if there is a prebuilt wheel, everything works perfect, but if there isn't a prebuilt wheel the fall back is to trigger the source build in a way that isn't recommended on Windows. This behavior seems odd to me.

That being said. There are clear instructions on how to do the build the right way, so maybe this behavior isn't an actual problem and this issue should be closed.

[radarhere]

Fyi, PyPy3.9 has now been released, and #6782 has added prebuilt wheels for Windows for the next Pillow release.