Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python 3.12 support with the xlclang compiler #1177

Closed
pfgdrupalas opened this issue Mar 14, 2024 · 18 comments
Closed

Python 3.12 support with the xlclang compiler #1177

pfgdrupalas opened this issue Mar 14, 2024 · 18 comments

Comments

@pfgdrupalas
Copy link

Hello,

We are needing to migrate to Python 3.12, but in trying to install JayDeBeapi, it looks like we are taking an error that is due to JPYPE not being supported for v3r12.

The install for JayDeBE looks like:
`Using pip 23.2.1 from /MVST/products/python/v3r12/pyz/lib/python3.12/site-packages/pip (python 3.12)
Collecting JayDeBeApi
Obtaining dependency information for JayDeBeApi from https://files.pythonhosted.org/packages/ff/1f/6a627c9bd7dea13235b65fce0fff987507269d41f957c578031796f70319/JayDeBeApi-1.2.3-py3-none-any.whl.metadata
Downloading JayDeBeApi-1.2.3-py3-none-any.whl.metadata (10 kB)
Collecting JPype1 (from JayDeBeApi)
Using cached JPype1-1.5.0.tar.gz (819 kB)
Running command pip subprocess to install build dependencies
Collecting setuptools>=40.8.0
Obtaining dependency information for setuptools>=40.8.0 from https://files.pythonhosted.org/packages/92/e1/1c8bb3420105e70bdf357d57dd5567202b4ef8d27f810e98bb962d950834/setuptools-69.2.0-py3-none-any.whl.metadata
Downloading setuptools-69.2.0-py3-none-any.whl.metadata (6.3 kB)
Collecting wheel
Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/7d/cd/d7460c9a869b16c3dd4e1e403cce337df165368c71d6af229a74699622ce/wheel-0.43.0-py3-none-any.whl.metadata
Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Downloading setuptools-69.2.0-py3-none-any.whl (821 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 821.5/821.5 kB 4.7 MB/s eta 0:00:00
Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
Installing collected packages: wheel, setuptools
Successfully installed setuptools-69.2.0 wheel-0.43.0

[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: python3.12 -m pip install --upgrade pip
Installing build dependencies ... done
Running command Getting requirements to build wheel
Falling back to provided JNI headers, since your provided JAVA_HOME "/usr/lpp/java/J17/J17.0_64" does not provide jni.h
running egg_info
writing JPype1.egg-info/PKG-INFO
writing dependency_links to JPype1.egg-info/dependency_links.txt
writing entry points to JPype1.egg-info/entry_points.txt
writing requirements to JPype1.egg-info/requires.txt
writing top-level names to JPype1.egg-info/top_level.txt
reading manifest file 'JPype1.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.class' under directory 'native'
no previously-included directories found matching 'test/classes/'
adding license file 'LICENSE'
adding license file 'NOTICE'
adding license file 'AUTHORS.rst'
writing manifest file 'JPype1.egg-info/SOURCES.txt'
Getting requirements to build wheel ... done
Running command Preparing metadata (pyproject.toml)
Falling back to provided JNI headers, since your provided JAVA_HOME "/usr/lpp/java/J17/J17.0_64" does not provide jni.h
running dist_info
creating /MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info
writing /MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/PKG-INFO
writing dependency_links to /MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/dependency_links.txt
writing entry points to /MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/entry_points.txt
writing requirements to /MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/requires.txt
writing top-level names to /MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/top_level.txt
writing manifest file '/MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/SOURCES.txt'
reading manifest file '/MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.class' under directory 'native'
no previously-included directories found matching 'test/classes/'
adding license file 'LICENSE'
adding license file 'NOTICE'
adding license file 'AUTHORS.rst'
writing manifest file '/MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1.egg-info/SOURCES.txt'
creating '/MVST/products/python/tmp/pip-modern-metadata-fz0_doqh/JPype1-1.5.0.dist-info'
Preparing metadata (pyproject.toml) ... done
Collecting packaging (from JPype1->JayDeBeApi)
Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl.metadata
Downloading packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Using cached JayDeBeApi-1.2.3-py3-none-any.whl (26 kB)
Downloading packaging-24.0-py3-none-any.whl (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 1.1 MB/s eta 0:00:00
Building wheels for collected packages: JPype1
Running command Building wheel for JPype1 (pyproject.toml)
Falling back to provided JNI headers, since your provided JAVA_HOME "/usr/lpp/java/J17/J17.0_64" does not provide jni.h
running bdist_wheel
running build
running build_py
creating build
creating build/lib.os390-28.00-3931-cpython-312
creating build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/init.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_classpath.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_core.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_gui.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jarray.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jclass.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jcollection.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jcustomizer.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jexception.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jinit.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jio.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jmethod.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jobject.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jpackage.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jproxy.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jstring.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jthread.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jvmfinder.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_pykeywords.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/beans.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/config.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/dbapi2.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/imports.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/nio.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/pickle.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/protocol.py -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/types.py -> build/lib.os390-28.00-3931-cpython-312/jpype
creating build/lib.os390-28.00-3931-cpython-312/jpype/_pyinstaller
copying jpype/_pyinstaller/entry_points.py -> build/lib.os390-28.00-3931-cpython-312/jpype/_pyinstaller
copying jpype/_pyinstaller/example.py -> build/lib.os390-28.00-3931-cpython-312/jpype/_pyinstaller
copying jpype/_pyinstaller/hook-jpype.py -> build/lib.os390-28.00-3931-cpython-312/jpype/_pyinstaller
copying jpype/_pyinstaller/test_jpype_pyinstaller.py -> build/lib.os390-28.00-3931-cpython-312/jpype/_pyinstaller
copying jpype/_jcollection.pyi -> build/lib.os390-28.00-3931-cpython-312/jpype
copying jpype/_jio.pyi -> build/lib.os390-28.00-3931-cpython-312/jpype
running build_ext
Call build extensions
Using Jar cache
copying native/jars/org.jpype.jar -> build/lib.os390-28.00-3931-cpython-312
Call build ext
building '_jpype' extension
creating build/temp.os390-28.00-3931-cpython-312
creating build/temp.os390-28.00-3931-cpython-312/native
creating build/temp.os390-28.00-3931-cpython-312/native/common
creating build/temp.os390-28.00-3931-cpython-312/native/python
/usr/lpp/cbclib/xlclang/bin/xlclang -DNDEBUG -D_XOPEN_SOURCE_EXTENDED -D_UNIX03_THREADS -D_POSIX_THREADS -D_OPEN_SYS_FILE_EXT -Inative/common/include -Inative/python/include -Inative/embedded/include -I/usr/lpp/java/J17/J17.0_64/include -Inative/jni_include -I/MVST/products/python/v3r12/pyz/include/python3.12 -c native/common/jp_array.cpp -o build/temp.os390-28.00-3931-cpython-312/native/common/jp_array.o
In file included from native/common/jp_array.cpp:16:
In file included from native/common/include/jpype.h:184:
In file included from native/python/include/jp_pythontypes.h:18:
In file included from /MVST/products/python/v3r12/pyz/include/python3.12/Python.h:38:
In file included from /MVST/products/python/v3r12/pyz/include/python3.12/pyport.h:202:
In file included from /usr/include/c++/math.h:301:
/usr/include/le/math.h:5327:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5361:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5393:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5426:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5459:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5492:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5529:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5562:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5595:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5631:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5664:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5697:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
/usr/include/le/math.h:5732:9: error: use of undeclared identifier '__static_assert'
__static_assert((sizeof(T) < 0 ),
^
13 errors generated.
Error while processing native/common/jp_array.cpp.
CCN0793(I) Compilation failed for file native/common/jp_array.cpp. Object file not created.
error: command '/usr/lpp/cbclib/xlclang/bin/xlclang' failed with exit code 12
error: subprocess-exited-with-error

× Building wheel for JPype1 (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /MVST/products/python/v3r12/pyz/bin/python3.12 /MVST/products/python/v3r12/pyz/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /products/python/tmp/tmpnbsasrnh
cwd: /MVST/products/python/tmp/pip-install-xh_2_l8i/jpype1_bf4998e581a14076b5a3a7c7e066b47b
Building wheel for JPype1 (pyproject.toml) ... error
ERROR: Failed building wheel for JPype1
Failed to build JPype1
ERROR: Could not build wheels for JPype1, which is required to install pyproject.toml-based projects
`

Thanks!
@Thrameos
Copy link
Contributor

#1147

The error message says something is wrong with math.h which is not part of jpype.

@pfgdrupalas
Copy link
Author

The end error:
Building wheel for JPype1 (pyproject.toml) ... error
ERROR: Failed building wheel for JPype1
Failed to build JPype1
ERROR: Could not build wheels for JPype1, which is required to install pyproject.toml-based projects

Indicates this was part of the JPYPE1 wheel build.

You can also see the program call error path:
In file included from native/common/jp_array.cpp:16:
In file included from native/common/include/jpype.h:184:
In file included from native/python/include/jp_pythontypes.h:18:
In file included from /MVST/products/python/v3r12/pyz/include/python3.12/Python.h:38:
In file included from /MVST/products/python/v3r12/pyz/include/python3.12/pyport.h:202:
In file included from /usr/include/c++/math.h:301:

Thanks!

@Thrameos
Copy link
Contributor

Yes... If JPype can't import the system include file by importing "Python.h" then it can't be built. But this has little to do with JPype and more to do with a broken system. Calling #include "Python.h" and having it fail is not a coding bug with the module, but rather an issue with the system.

I see a few possibilities:

  1. The flags required for your system are different that others. You would need to figure out what those flags need to be and modify "setupext/platform.py" to reflect the changes and submit a PR.
  2. Your system is just busted with a C++ compiler and headers that are incompatible. Meaning any C++ code that includes Python.h will bust. In such a case, the error has nothing to do with JPype.

If I had to guess the headers for C++ and the compiler came from different packages and that is causing the issue. But I can't rule out the flags. Just doing a google search, see page 13 of https://www.ibm.com/support/pages/sites/default/files/inline-files/$FILE/SC31580100_0.pdf

@pfgdrupalas
Copy link
Author

pfgdrupalas commented Mar 16, 2024 via email

@Thrameos
Copy link
Contributor

@pelson could there be a change in the flags with the installion upgrade? I don't see how it would work in 3.9 and not 3.12. As the error is in a system library, the only way I could see a difference is if Python 3.12 shipped with flags that were incompatible with c++. Unfortunately I don't know xlclang, but it does seem odd that the c++11 flag isn't there, so I am hoping you may have an idea.

@pelson
Copy link
Contributor

pelson commented Mar 18, 2024

The first thing to say - JPype does support Python 3.12, and even ships pre-built binaries for the manylinux platforms. The JPype pipelines validate Python 3.12 support systematically. I think we should update the issue's title to be more helpful - "Python 3.12 support with the xlclang compiler". Unfortunately, I don't have the permission to do that (perhaps that is something we can fix @Thrameos / @marscher?).

It seems unlikely to me that this has changed with the move to pyproject.toml in !1133, for a start, this report is based on a build coming from a source tarball release from December 😉.

As for being able to reproduce this - I don't have access to an machine with xlclang either.

We are just trying to migrate to v3.12 and took this type of error on 3 of the 13 install items.

Is this to say that this isn't JPype specific, and you are seeing it with other project too? In such a scenario it is clear that the problem is that the system is not in a good state, and that is not something we can fix/workaround on the JPype level.

@pfgdrupalas pfgdrupalas changed the title When will JPYPE support v3.12? Python 3.12 support with the xlclang compiler Mar 18, 2024
@pfgdrupalas
Copy link
Author

Hello,

The 3 modules that failed xlclang are JayDeBe (install log provided), ansible-tower, and zoautil_py,

The rest did not try to use/need xlclang.

Other than the versions of the modules trying to be installed nothing has change on our system. We are running on an IBM z/OS mainframe.

So, yes something might be "off" with our system, but prior versions of these modules installed just fine with no errors, so I am at a loss as to how to try and figure out what exactly is wrong.

Thanks!

@pelson
Copy link
Contributor

pelson commented Mar 18, 2024

Are you able to re-build JPype on Python 3.9? At least then we can see if this is a indeed something that has changed on our side, or if this is a system change for which you will need to speak to your sys admins about.

@pfgdrupalas
Copy link
Author

pfgdrupalas commented Mar 18, 2024 •
edited
Loading

Hello,

Yes, I was able to uninstall and then reinstall JPYPE1 in our Python 3.9 environment:

pip3 install --trusted-host pypi.org --trusted-host files.pythonhosted.org JPype1
Collecting JPype1
Using cached JPype1-1.5.0.tar.gz (819 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: packaging in /MVST/products/python/v3r9/pyz/lib/python3.9/site-packages (from JPype1) (23.1)
Building wheels for collected packages: JPype1
Building wheel for JPype1 (pyproject.toml) ... done
Created wheel for JPype1: filename=JPype1-1.5.0-cp39-cp39-os390_28_00_3931.whl size=1559027 sha256=5ed06b0fa8f957b1a8e0f9a2f3ef6d26fa8578e0120a530c9fc6aaa22ab13cf3
Stored in directory: /u/d020216/.cache/pip/wheels/25/db/5e/17e495c39f5d36df1078c226f875a890ccbf1ead1b243524d8
Successfully built JPype1
Installing collected packages: JPype1
Successfully installed JPype1-1.5.0

pip3 list
Package Version

...
Jinja2 3.1.2
JPype1 1.5.0
...
wheel 0.40.0
...

The version of JPype1 was 1.4.1, but pulling in new got us to 1.5.0

Thanks!

@Thrameos
Copy link
Contributor

Thrameos commented Mar 18, 2024 via email

@pfgdrupalas
Copy link
Author

pfgdrupalas commented Mar 18, 2024 via email

@marscher
Copy link
Member

you need to clone this repository (or dowload the code as archive and unpack) to run the specified command.

@Thrameos
Copy link
Contributor

Thrameos commented Mar 18, 2024 via email

@pfgdrupalas
Copy link
Author

pfgdrupalas commented Mar 18, 2024 •
edited
Loading

For Python 3.9, that line is:
/usr/lpp/cbclib/xlclang/bin/xlclang++ -fno-strict-aliasing -DNDEBUG -qarch=10 -q64 -Wc,DLL -D_XOPEN_SOURCE_EXTENDED -D_UNIX03_THREADS -D_POSIX_THREADS -D_OPEN_SYS_FILE_EXT -qexportall -qascii -qstrict -qnocsect -Wa,asa,goff -Wa,xplink -qgonumber -qenum=int -Inative/common/include -Inative/python/include -Inative/embedded/include -I/usr/lpp/java/J11/J11.0_64/include -I/usr/lpp/java/J11/J11.0_64/include/zos -I/MVST/products/python/v3r9/pyz/include/python3.9 -c native/common/jp_array.cpp -o build/temp.os390-28.00-3931-3.9/native/common/jp_array.o

And for Python 3.12 the line is:
/usr/lpp/cbclib/xlclang/bin/xlclang -DNDEBUG -D_XOPEN_SOURCE_EXTENDED -D_UNIX03_THREADS -D_POSIX_THREADS -D_OPEN_SYS_FILE_EXT -Inative/common/include -Inative/python/include -Inative/embedded/include -I/usr/lpp/java/J11/J11.0_64/include -I/usr/lpp/java/J11/J11.0_64/include/zos -I/MVST/products/python/v3r12/pyz/include/python3.12 -c native/common/jp_array.cpp -o build/temp.os390-28.00-3931-cpython-312/native/common/jp_array.o

Thanks!

@Thrameos
Copy link
Contributor

Thrameos commented Mar 18, 2024 via email

@pfgdrupalas
Copy link
Author

pfgdrupalas commented Mar 18, 2024 via email

@Thrameos
Copy link
Contributor

Thrameos commented Mar 20, 2024 via email

@Thrameos
Copy link
Contributor

Thrameos commented Mar 24, 2024 •
edited
Loading

Here is the code to check your compiler settings

from setuptools._distutils import sysconfig

print({p:v for p,v in sysconfig.get_config_vars().items() if isinstance(v,str) and "CC" in p})
print({p:v for p,v in sysconfig.get_config_vars().items() if isinstance(v,str) and "CXX" in p})

This should print out your C and C++ compiler settings.

For my system I get something like this

{'CC': 'x86_64-linux-gnu-gcc', 'CCSHARED': '-fPIC', 'LINKCC': 'x86_64-linux-gnu-gcc', 'MAINCC': 'x86_64-linux-gnu-gcc'}
{'CXX': 'x86_64-linux-gnu-g++', 'LDCXXSHARED': 'x86_64-linux-gnu-g++ -shared -Wl,-O1 -Wl,-Bsymbolic-functions'}

You need to report to your system admin if the CXX and LDCXXSHARED are reporting C rather than C++ versions of the compiler.

I will close this issue as this isn't a problem with JPype but rather a site issue with a build of Python, so it doesn't really apply to anyone else. You can continue to post to this thread and I will respond.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@marscher @pelson @pfgdrupalas @Thrameos