Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation fault when running tests in python-3.10 with 0.7.a2 and 0.7.0a3, 0.6.0 is fine #165

Closed
edgarcosta opened this issue Jul 21, 2024 · 16 comments · Fixed by #171
Closed

Comments

@edgarcosta
Copy link
Member

edgarcosta commented Jul 21, 2024

Here is an edited log.

I couldn't try 0.7.a1, as it fails to find a wheel, and to build, as expected.

edgarcosta@diophantus ~/test ⚡ source bin/activate
(test) edgarcosta@diophantus ~/test ⚡ pip3 install python-flint
Collecting python-flint
  Downloading python_flint-0.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (41.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.1/41.1 MB 18.6 MB/s eta 0:00:00
Installing collected packages: python-flint
Successfully installed python-flint-0.6.0
(test) edgarcosta@diophantus ~/test ⚡ python3 -m flint.test
Running tests...
[...]
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------
(test) edgarcosta@diophantus ~/test ⚡ pip3 install python-flint==0.7.0a1 # it can't build, as expected
Collecting python-flint==0.7.0a1
  Downloading python_flint-0.7.0a1.tar.gz (229 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 KB 4.5 MB/s eta 0:00:00
[...]
      ../meson.build:10:12: ERROR: Dependency "flint" not found, tried pkgconfig and make
[...]
(test) edgarcosta@diophantus ~/test 💩⚡ pip3 install python-flint==0.7.0a2
Collecting python-flint==0.7.0a2
  Downloading python_flint-0.7.0a2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 29.5 MB/s eta 0:00:00
Installing collected packages: python-flint
  Attempting uninstall: python-flint
    Found existing installation: python-flint 0.6.0
    Uninstalling python-flint-0.6.0:
      Successfully uninstalled python-flint-0.6.0
Successfully installed python-flint-0.7.0a2
(test) edgarcosta@diophantus ~/test ⚡ python3 -m flint.test
Running tests...
test_pyflint...Illegal instruction (core dumped)
(test) edgarcosta@diophantus ~/test 💩⚡ pip3 install python-flint==0.7.0a3
Collecting python-flint==0.7.0a3
  Using cached python_flint-0.7.0a3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.0 MB)
Installing collected packages: python-flint
  Attempting uninstall: python-flint
    Found existing installation: python-flint 0.7.0a2
    Uninstalling python-flint-0.7.0a2:
      Successfully uninstalled python-flint-0.7.0a2
Successfully installed python-flint-0.7.0a3
(test) edgarcosta@diophantus ~/test ⚡ python3 -m flint.test
Running tests...
test_pyflint...Illegal instruction (core dumped)
(test) edgarcosta@diophantus ~/test 💩⚡ python3
Python 3.10.12 (main, Mar 22 2024, 16:50:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
@oscarbenjamin
Copy link
Collaborator

That's worrying. I can't reproduce this on MacOS. I don't immediately have Linux available to test:

$ pip install python-flint==0.7.0a3
...
$ python -m flint.test
...
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

(test) edgarcosta@diophantus ~/test ⚡ python3 -m flint.test
Running tests...
test_pyflint...Illegal instruction (core dumped)

I wonder if we are building the libraries for generic x86_64 correctly:

curl -O -L https://github.com/flintlib/flint/releases/download/v$FLINTVER/flint-$FLINTVER.tar.gz
tar xf flint-$FLINTVER.tar.gz
cd flint-$FLINTVER
./bootstrap.sh
./configure --prefix=$PREFIX\
$FLINTARB_WITHGMP\
--with-mpfr=$PREFIX\
--disable-static\
--disable-debug
make -j6
make install
cd ..

Maybe that needs --host=$HOSTARG to get the generic host CPU option:

yum install -y xz
bin/build_dependencies_unix.sh\
--gmp gmp\
--host x86_64-pc-linux-gnu\
--use-gmp-github-mirror

What does Flint's config.guess report on your system?

@oscarbenjamin
Copy link
Collaborator

This is in the ./configure output for Flint as built in CI

2024-07-10T01:09:05.4877844Z checking for host architecture... znver2
2024-07-10T01:09:05.6506050Z checking for immintrin.h... yes
2024-07-10T01:09:05.6576656Z checking if system have required x86_64 instruction set for fft_small... yes
2024-07-10T01:09:05.6578687Z checking if system can use FLINT's fft_small module... yes
2024-07-10T01:09:05.6647828Z checking if system uses System V ABI... yes
2024-07-10T01:09:05.6714235Z checking if ADX instruction set is supported by CPU... yes
2024-07-10T01:09:05.6715060Z checking if system can use FLINT's assembly... yes

https://github.com/flintlib/python-flint/actions/runs/9866479989/job/27246112789

We need to prevent Flint from detecting the CPU in the build machine. It also looks like we should set the --host flag for mpfr. It already is used for GMP.

Is there something about this that has changed in recent Flint versions e.g. 3.1.0 or something? I haven't seen any problems with python-flint 0.6.0 which is built using Flint 3.0.1.

Now with a greater understanding of Flint's build configuration I can see what all of this is doing. Full log:

2024-07-10T01:09:03.5959675Z checking build system type... x86_64-pc-linux-gnu
2024-07-10T01:09:03.5960233Z checking host system type... x86_64-pc-linux-gnu
2024-07-10T01:09:03.5973971Z checking how to print strings... printf
2024-07-10T01:09:03.5979183Z checking for gcc... gcc
2024-07-10T01:09:03.6407777Z checking whether the C compiler works... yes
2024-07-10T01:09:03.6408516Z checking for C compiler default output file name... a.out
2024-07-10T01:09:03.6652741Z checking for suffix of executables... 
2024-07-10T01:09:03.6947169Z checking whether we are cross compiling... no
2024-07-10T01:09:03.7097359Z checking for suffix of object files... o
2024-07-10T01:09:03.7236993Z checking whether the compiler supports GNU C... yes
2024-07-10T01:09:03.7393562Z checking whether gcc accepts -g... yes
2024-07-10T01:09:03.7780160Z checking for gcc option to enable C11 features... none needed
2024-07-10T01:09:03.7802340Z checking for a sed that does not truncate output... /usr/bin/sed
2024-07-10T01:09:03.7826870Z checking for grep that handles long lines and -e... /usr/bin/grep
2024-07-10T01:09:03.7839578Z checking for egrep... /usr/bin/grep -E
2024-07-10T01:09:03.7852289Z checking for fgrep... /usr/bin/grep -F
2024-07-10T01:09:03.7894424Z checking for ld used by gcc... /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld
2024-07-10T01:09:03.7917924Z checking if the linker (/opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld) is GNU ld... yes
2024-07-10T01:09:03.7927442Z checking for BSD- or MS-compatible name lister (nm)... /opt/rh/devtoolset-10/root/usr/bin/nm -B
2024-07-10T01:09:03.8071782Z checking the name lister (/opt/rh/devtoolset-10/root/usr/bin/nm -B) interface... BSD nm
2024-07-10T01:09:03.8073309Z checking whether ln -s works... yes
2024-07-10T01:09:03.8102186Z checking the maximum length of command line arguments... 3145728
2024-07-10T01:09:03.8124362Z checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
2024-07-10T01:09:03.8127494Z checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
2024-07-10T01:09:03.8128580Z checking for /opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld option to reload object files... -r
2024-07-10T01:09:03.8138556Z checking for file... file
2024-07-10T01:09:03.8145732Z checking for objdump... objdump
2024-07-10T01:09:03.8152747Z checking how to recognize dependent libraries... pass_all
2024-07-10T01:09:03.8162751Z checking for dlltool... no
2024-07-10T01:09:03.8164277Z checking how to associate runtime and link libraries... printf %s\n
2024-07-10T01:09:03.8169099Z checking for ar... ar
2024-07-10T01:09:03.8376564Z checking for archiver @FILE support... @
2024-07-10T01:09:03.8380866Z checking for strip... strip
2024-07-10T01:09:03.8387381Z checking for ranlib... ranlib
2024-07-10T01:09:03.8392981Z checking for gawk... gawk
2024-07-10T01:09:03.8888613Z checking command to parse /opt/rh/devtoolset-10/root/usr/bin/nm -B output from gcc object... ok
2024-07-10T01:09:03.8903827Z checking for sysroot... no
2024-07-10T01:09:03.8938867Z checking for a working dd... /usr/bin/dd
2024-07-10T01:09:03.8962724Z checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
2024-07-10T01:09:03.9079063Z checking for mt... no
2024-07-10T01:09:03.9101027Z checking if : is a manifest tool... no
2024-07-10T01:09:03.9259862Z checking for stdio.h... yes
2024-07-10T01:09:03.9442670Z checking for stdlib.h... yes
2024-07-10T01:09:03.9652324Z checking for string.h... yes
2024-07-10T01:09:03.9872670Z checking for inttypes.h... yes
2024-07-10T01:09:04.0075011Z checking for stdint.h... yes
2024-07-10T01:09:04.0295329Z checking for strings.h... yes
2024-07-10T01:09:04.0522045Z checking for sys/stat.h... yes
2024-07-10T01:09:04.0739697Z checking for sys/types.h... yes
2024-07-10T01:09:04.0978773Z checking for unistd.h... yes
2024-07-10T01:09:04.1220053Z checking for dlfcn.h... yes
2024-07-10T01:09:04.1247835Z checking for objdir... .libs
2024-07-10T01:09:04.1830817Z checking if gcc supports -fno-rtti -fno-exceptions... no
2024-07-10T01:09:04.1833494Z checking for gcc option to produce PIC... -fPIC -DPIC
2024-07-10T01:09:04.1980088Z checking if gcc PIC flag -fPIC -DPIC works... yes
2024-07-10T01:09:04.2151944Z checking if gcc static flag -static works... no
2024-07-10T01:09:04.2346329Z checking if gcc supports -c -o file.o... yes
2024-07-10T01:09:04.2347295Z checking if gcc supports -c -o file.o... (cached) yes
2024-07-10T01:09:04.2420518Z checking whether the gcc linker (/opt/rh/devtoolset-10/root/usr/libexec/gcc/x86_64-redhat-linux/10/ld -m elf_x86_64) supports shared libraries... yes
2024-07-10T01:09:04.2684449Z checking whether -lc should be explicitly linked in... no
2024-07-10T01:09:04.3078525Z checking dynamic linker characteristics... GNU/Linux ld.so
2024-07-10T01:09:04.3079643Z checking how to hardcode library paths into programs... immediate
2024-07-10T01:09:04.3093806Z checking whether stripping libraries is possible... yes
2024-07-10T01:09:04.3094684Z checking if libtool supports shared libraries... yes
2024-07-10T01:09:04.3099316Z checking whether to build shared libraries... yes
2024-07-10T01:09:04.3100038Z checking whether to build static libraries... no
2024-07-10T01:09:04.3216775Z checking for a race-free mkdir -p... /usr/bin/mkdir -p
2024-07-10T01:09:04.3314929Z checking for inline... inline
2024-07-10T01:09:04.3753578Z checking whether byte ordering is bigendian... no
2024-07-10T01:09:04.3764046Z checking if memory is strongly-ordered... yes
2024-07-10T01:09:04.3925052Z checking how to run the C preprocessor... gcc -E
2024-07-10T01:09:04.4316335Z checking for gmp.h... yes
2024-07-10T01:09:04.4399572Z checking if version of GMP is greater than 6.2.1... yes
2024-07-10T01:09:04.4525730Z checking if GMP defines mp_limb_t as unsigned long long int... no
2024-07-10T01:09:04.4773789Z checking for mpfr.h... yes
2024-07-10T01:09:04.4866426Z checking if version of MPFR is greater than 4.1.0... yes
2024-07-10T01:09:04.4959712Z checking for desired ABI... 64
2024-07-10T01:09:04.5165681Z checking for stdarg.h... yes
2024-07-10T01:09:04.5442968Z checking for math.h... yes
2024-07-10T01:09:04.5669905Z checking for float.h... yes
2024-07-10T01:09:04.5903454Z checking for errno.h... yes
2024-07-10T01:09:04.6117591Z checking for fenv.h... yes
2024-07-10T01:09:04.6318455Z checking for alloca.h... yes
2024-07-10T01:09:04.6513287Z checking for malloc.h... yes
2024-07-10T01:09:04.6736294Z checking for sys/param.h... yes
2024-07-10T01:09:04.6931819Z checking for windows.h... no
2024-07-10T01:09:04.7129365Z checking for pthread_np.h... no
2024-07-10T01:09:04.7347746Z checking for pthread.h... yes
2024-07-10T01:09:04.7922178Z checking for library containing atan2... -lm
2024-07-10T01:09:04.8503287Z checking for library containing __gmpz_init... -lgmp
2024-07-10T01:09:04.8793241Z checking for library containing __gmpn_gcd_11... none required
2024-07-10T01:09:04.9083675Z checking for library containing __gmpn_div_q... none required
2024-07-10T01:09:04.9382249Z checking for library containing __gmpn_mul_basecase... none required
2024-07-10T01:09:05.0014779Z checking for library containing __gmpn_modexact_1_odd... no
2024-07-10T01:09:05.0308910Z checking for library containing __gmpn_addmul_2... none required
2024-07-10T01:09:05.0919676Z checking for library containing mpfr_init... -lmpfr
2024-07-10T01:09:05.1222647Z checking for library containing mpfr_round_p... none required
2024-07-10T01:09:05.1527610Z checking for library containing mpfr_mulhigh_n... none required
2024-07-10T01:09:05.1838684Z checking for library containing mpfr_sqrhigh_n... none required
2024-07-10T01:09:05.1982094Z checking whether gcc accepts -pthread... yes
2024-07-10T01:09:05.2332092Z checking if cpu_set_t is supported... yes
2024-07-10T01:09:05.2648600Z checking if alloca works... yes
2024-07-10T01:09:05.3007571Z checking for aligned_alloc... yes
2024-07-10T01:09:05.3358973Z checking for _aligned_malloc... no
2024-07-10T01:09:05.3509208Z checking whether gcc accepts -Werror=unknown-warning-option... no
2024-07-10T01:09:05.3655915Z checking whether gcc accepts -Wall... yes
2024-07-10T01:09:05.3795365Z checking whether gcc accepts -Wno-stringop-overread... yes
2024-07-10T01:09:05.3935617Z checking whether gcc accepts -Wno-stringop-overflow... yes
2024-07-10T01:09:05.4076813Z checking whether gcc accepts -Werror=implicit-function-declaration... yes
2024-07-10T01:09:05.4223788Z checking whether gcc accepts -O3... yes
2024-07-10T01:09:05.4382182Z checking whether gcc accepts -march=native... yes
2024-07-10T01:09:05.4518477Z checking whether gcc accepts -std=c11... yes
2024-07-10T01:09:05.4658324Z checking whether gcc accepts -pedantic... yes
2024-07-10T01:09:05.4877844Z checking for host architecture... znver2
2024-07-10T01:09:05.6506050Z checking for immintrin.h... yes
2024-07-10T01:09:05.6576656Z checking if system have required x86_64 instruction set for fft_small... yes
2024-07-10T01:09:05.6578687Z checking if system can use FLINT's fft_small module... yes
2024-07-10T01:09:05.6647828Z checking if system uses System V ABI... yes
2024-07-10T01:09:05.6714235Z checking if ADX instruction set is supported by CPU... yes
2024-07-10T01:09:05.6715060Z checking if system can use FLINT's assembly... yes
2024-07-10T01:09:05.6767744Z checking for suitable m4... m4
2024-07-10T01:09:05.6833394Z checking how to switch to text section... .text
2024-07-10T01:09:05.6834167Z checking how to switch to data section... .data
2024-07-10T01:09:05.6962531Z checking for assembler label suffix... :
2024-07-10T01:09:05.6963205Z checking for assembler global directive... .globl
2024-07-10T01:09:05.6963775Z checking for assembler global directive attribute... 
2024-07-10T01:09:05.7083577Z checking if globals are prefixed by underscore... no
2024-07-10T01:09:05.7187375Z checking how to switch to read-only data section... 	.section	.rodata
2024-07-10T01:09:05.7265746Z checking for assembler .type directive... .type	$1,@$2
2024-07-10T01:09:05.7341387Z checking for assembler .size directive... .size	$1,$2
2024-07-10T01:09:05.7523956Z checking for assembler local label prefix... .L
2024-07-10T01:09:05.7582157Z checking for assembler byte directive... .byte
2024-07-10T01:09:05.7665361Z checking if .align assembly directive is logarithmic... no
2024-07-10T01:09:05.7743671Z checking if the .align directive accepts an 0x90 fill in .text... yes
2024-07-10T01:09:05.7817611Z checking for assembler COFF type directives... no
2024-07-10T01:09:05.7818367Z creating config.m4
2024-07-10T01:09:05.8040306Z configure: creating ./config.status

@oscarbenjamin
Copy link
Collaborator

Edgar can you try downloading the Ubuntu wheels artefact from the bottom of the actions page here:
https://github.com/flintlib/python-flint/actions/runs/10026947402?pr=166

@oscarbenjamin
Copy link
Collaborator

Actually maybe setting --host is not enough. We still seem to have:

2024-07-21T08:35:27.1413568Z checking for host architecture... znver2
2024-07-21T08:35:27.3209306Z checking for immintrin.h... yes
2024-07-21T08:35:27.3310780Z checking if system have required x86_64 instruction set for fft_small... yes
2024-07-21T08:35:27.3312315Z checking if system can use FLINT's fft_small module... yes
2024-07-21T08:35:27.3416055Z checking if system uses System V ABI... yes
2024-07-21T08:35:27.3517812Z checking if ADX instruction set is supported by CPU... yes
2024-07-21T08:35:27.3518740Z checking if system can use FLINT's assembly... yes
2024-07-21T08:35:27.3587163Z checking for suitable m4... m4
2024-07-21T08:35:27.3681650Z checking how to switch to text section... .text
2024-07-21T08:35:27.3682652Z checking how to switch to data section... .data
2024-07-21T08:35:27.3883114Z checking for assembler label suffix... :
2024-07-21T08:35:27.3883923Z checking for assembler global directive... .globl
2024-07-21T08:35:27.3884712Z checking for assembler global directive attribute... 
2024-07-21T08:35:27.4071489Z checking if globals are prefixed by underscore... no
2024-07-21T08:35:27.4226008Z checking how to switch to read-only data section... 	.section	.rodata
2024-07-21T08:35:27.4329056Z checking for assembler .type directive... .type	$1,@$2
2024-07-21T08:35:27.4431729Z checking for assembler .size directive... .size	$1,$2
2024-07-21T08:35:27.4704111Z checking for assembler local label prefix... .L
2024-07-21T08:35:27.4797411Z checking for assembler byte directive... .byte
2024-07-21T08:35:27.4920257Z checking if .align assembly directive is logarithmic... no
2024-07-21T08:35:27.5022479Z checking if the .align directive accepts an 0x90 fill in .text... yes

@oscarbenjamin
Copy link
Collaborator

What is the correct way to build Flint if the intention is to distribute a generic x86_64 binary?

@oscarbenjamin
Copy link
Collaborator

I've attached three different zip files containing manylinux x86-64 wheels for CPython 3.10. I don't have an x86-64 system available to test right now. Can you test installing each of these wheels to see if any work:

wheel_flint_3.0.1.zip
wheel_host.zip
wheel_no_assembly.zip

You should be able to just pip install ./filename.whl.

I think that maybe something has changed in Flint 3.1 that means that our CI wheel build no longer produces generic x86-64 binaries on Linux. We will need to find a way to build generic binaries with Flint 3.1 or otherwise drop back to shipping Flint 3.0.1 in the PyPI wheels.

Previous releases of python-flint have used Flint < 3.1 and have not seen this problem.

@edgarcosta
Copy link
Member Author

For the record, this is how I enabled generic bottle for flint in homebrew:
https://github.com/Homebrew/homebrew-core/blob/680b728a0761aa8df5afed82afbffd102b2caaea/Formula/f/flint.rb#L49

@edgarcosta
Copy link
Member Author

@edgarcosta
Copy link
Member Author

I will try to build my own wheels on a vm, and see if I can tweak it to make it work

@oscarbenjamin
Copy link
Collaborator

I will try to build my own wheels on a vm, and see if I can tweak it to make it work

The wheels I showed above are all built in the CI here. If you open a pull request then you can download the wheels from the GitHub Actions artifacts. That's the best way to test because that is also how the wheels are made for deployment to PyPI.

@oscarbenjamin
Copy link
Collaborator

I've bumped the version in 0.7.0a4 ready to push a new release to PyPI. Once we can test that installing from PyPI works we can close this.

@oscarbenjamin
Copy link
Collaborator

The 0.7.0a4 release is on pypi now so hopefully this is fixed. We will need to remember when Flint 3.2 comes around to change the build flags (remove --enable-arch). We can close this once you confirm that installing from pypi works

@edgarcosta
Copy link
Member Author

it all works fine

@edgarcosta
Copy link
Member Author

ps: the --enable-arch will just be ignored in the new release, so there is no harm in having it there

@oscarbenjamin
Copy link
Collaborator

Thanks Edgar for reporting and testing/fixing this. The number one thing that worries me with python-flint is that it might fail on other people's computers and that I won't have access to the hardware to test it. I tried to make checks comprehensive in CI but something like this appears to pass in CI even though it will fail for actual users.

@edgarcosta
Copy link
Member Author

I understand; yeah, testing on old hardware is hard. Luckily, I have plenty of that around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants