BUG: Error compiling wheels using crossenv #1957

Soft-Buddy · 2024-02-28T17:16:43Z

This is a pyzmq bug

This is a pyzmq-specific bug, not an issue of zmq socket behavior. Don't worry if you're not sure! We'll figure it out together.

What pyzmq version?

25.1.2

What libzmq version?

4.3.5

Python version (and how it was installed)

python 3.11.8, cross compiled using ndk

OS

Host - Ubuntu

What happened?

I'm trying to cross compile pyzmq for Android, using crossenv, I'm running :

ZMQ_PREFIX=/home/rudra/Documents/libzmq-builds/arm64 LDFLAGS+="-L../lib/ -lpython3.11" CFLAGS="-I../include/python3.11" python setup.py bdist_wheel

Output :

(cross) rudra@rudra-1-2:~/Documents/compilation/aarch64/pyzmq$ ZMQ_PREFIX=/home/rudra/Documents/libzmq-builds/arm64 LDFLAGS+="-L../lib/ -lpython3.11" CFLAGS="-I../include/python3.11" python setup.py bdist_wheel
/home/rudra/Documents/compilation/aarch64/venv/build/lib/python3.11/site-packages/setuptools/_distutils/dist.py:265: UserWarning: Unknown distribution option: 'cffi_modules'
warnings.warn(msg)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-aarch64-cpython-311
creating build/lib.linux-aarch64-cpython-311/zmq
copying zmq/constants.py -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/error.py -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/decorators.py -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/_typing.py -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/init.py -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/_future.py -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/asyncio.py -> build/lib.linux-aarch64-cpython-311/zmq
creating build/lib.linux-aarch64-cpython-311/zmq/green
copying zmq/green/poll.py -> build/lib.linux-aarch64-cpython-311/zmq/green
copying zmq/green/device.py -> build/lib.linux-aarch64-cpython-311/zmq/green
copying zmq/green/core.py -> build/lib.linux-aarch64-cpython-311/zmq/green
copying zmq/green/init.py -> build/lib.linux-aarch64-cpython-311/zmq/green
creating build/lib.linux-aarch64-cpython-311/zmq/green/eventloop
copying zmq/green/eventloop/zmqstream.py -> build/lib.linux-aarch64-cpython-311/zmq/green/eventloop
copying zmq/green/eventloop/ioloop.py -> build/lib.linux-aarch64-cpython-311/zmq/green/eventloop
copying zmq/green/eventloop/init.py -> build/lib.linux-aarch64-cpython-311/zmq/green/eventloop
creating build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/devices/proxydevice.py -> build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/devices/basedevice.py -> build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/devices/monitoredqueue.py -> build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/devices/proxysteerabledevice.py -> build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/devices/monitoredqueuedevice.py -> build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/devices/init.py -> build/lib.linux-aarch64-cpython-311/zmq/devices
creating build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/interop.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/monitor.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/z85.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/garbage.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/strtypes.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/jsonapi.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/win32.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/init.py -> build/lib.linux-aarch64-cpython-311/zmq/utils
creating build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/context.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/poll.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/tracker.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/version.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/frame.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/socket.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/init.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/attrsettr.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/sugar/stopwatch.py -> build/lib.linux-aarch64-cpython-311/zmq/sugar
creating build/lib.linux-aarch64-cpython-311/zmq/ssh
copying zmq/ssh/tunnel.py -> build/lib.linux-aarch64-cpython-311/zmq/ssh
copying zmq/ssh/forward.py -> build/lib.linux-aarch64-cpython-311/zmq/ssh
copying zmq/ssh/init.py -> build/lib.linux-aarch64-cpython-311/zmq/ssh
creating build/lib.linux-aarch64-cpython-311/zmq/log
copying zmq/log/handlers.py -> build/lib.linux-aarch64-cpython-311/zmq/log
copying zmq/log/main.py -> build/lib.linux-aarch64-cpython-311/zmq/log
copying zmq/log/init.py -> build/lib.linux-aarch64-cpython-311/zmq/log
creating build/lib.linux-aarch64-cpython-311/zmq/auth
copying zmq/auth/base.py -> build/lib.linux-aarch64-cpython-311/zmq/auth
copying zmq/auth/certs.py -> build/lib.linux-aarch64-cpython-311/zmq/auth
copying zmq/auth/thread.py -> build/lib.linux-aarch64-cpython-311/zmq/auth
copying zmq/auth/ioloop.py -> build/lib.linux-aarch64-cpython-311/zmq/auth
copying zmq/auth/init.py -> build/lib.linux-aarch64-cpython-311/zmq/auth
copying zmq/auth/asyncio.py -> build/lib.linux-aarch64-cpython-311/zmq/auth
creating build/lib.linux-aarch64-cpython-311/zmq/backend
copying zmq/backend/select.py -> build/lib.linux-aarch64-cpython-311/zmq/backend
copying zmq/backend/init.py -> build/lib.linux-aarch64-cpython-311/zmq/backend
creating build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/context.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/utils.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/error.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/socket.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/message.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/_poll.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/devices.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cffi/init.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
creating build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/init.py -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
creating build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_decorators.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_monitor.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_monqueue.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_z85.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_asyncio.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_cython.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_draft.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_pubsub.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_mypy.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_cffi_backend.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_ext.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_socket.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_pair.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_context.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_error.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_includes.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_ssh.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_poll.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_proxy_steerable.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_multipart.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_log.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/conftest.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_message.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_imports.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_future.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_ioloop.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_zmqstream.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_retry_eintr.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_device.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_security.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_etc.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_constants.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_win32_shim.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_reqrep.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_version.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/test_auth.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
copying zmq/tests/init.py -> build/lib.linux-aarch64-cpython-311/zmq/tests
creating build/lib.linux-aarch64-cpython-311/zmq/eventloop
copying zmq/eventloop/_deprecated.py -> build/lib.linux-aarch64-cpython-311/zmq/eventloop
copying zmq/eventloop/zmqstream.py -> build/lib.linux-aarch64-cpython-311/zmq/eventloop
copying zmq/eventloop/future.py -> build/lib.linux-aarch64-cpython-311/zmq/eventloop
copying zmq/eventloop/ioloop.py -> build/lib.linux-aarch64-cpython-311/zmq/eventloop
copying zmq/eventloop/init.py -> build/lib.linux-aarch64-cpython-311/zmq/eventloop
copying zmq/init.pxd -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/init.pyi -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/py.typed -> build/lib.linux-aarch64-cpython-311/zmq
copying zmq/devices/monitoredqueue.pxd -> build/lib.linux-aarch64-cpython-311/zmq/devices
copying zmq/utils/buffers.pxd -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/pyversion_compat.h -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/ipcmaxlen.h -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/mutex.h -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/getpid_compat.h -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/zmq_compat.h -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/config.json -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/utils/compiler.json -> build/lib.linux-aarch64-cpython-311/zmq/utils
copying zmq/sugar/init.pyi -> build/lib.linux-aarch64-cpython-311/zmq/sugar
copying zmq/backend/init.pyi -> build/lib.linux-aarch64-cpython-311/zmq/backend
copying zmq/backend/cffi/_cdefs.h -> build/lib.linux-aarch64-cpython-311/zmq/backend/cffi
copying zmq/backend/cython/message.pxd -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/socket.pxd -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/checkrc.pxd -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/libzmq.pxd -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/init.pxd -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/context.pxd -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/backend/cython/constant_enums.pxi -> build/lib.linux-aarch64-cpython-311/zmq/backend/cython
copying zmq/tests/cython_ext.pyx -> build/lib.linux-aarch64-cpython-311/zmq/tests
running build_ext
running configure
{'libraries': ['zmq'], 'include_dirs': ['/home/rudra/Documents/libzmq-builds/arm64/include'], 'library_dirs': ['/home/rudra/Documents/libzmq-builds/arm64/lib'], 'runtime_library_dirs': ['/home/rudra/Documents/libzmq-builds/arm64/lib'], 'extra_link_args': []}

Configure: Autodetecting ZMQ settings...
Custom ZMQ dir: /home/rudra/Documents/libzmq-builds/arm64
Checking for timer_create
** Errors about missing timer_create are a normal part of this process **
/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../include/python3.11 -I/home/rudra/Documents/compilation/aarch64/venv/cross/include -I/usr/include/python3.11 -c /tmp/timer_createfylxmpzs.c -o tmp/timer_createfylxmpzs.o
/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang tmp/timer_createfylxmpzs.o -L/usr/lib -o a.out
/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I../include/python3.11 -I/home/rudra/Documents/libzmq-builds/arm64/include -Izmq/utils -c build/temp.linux-aarch64-cpython-311/scratch/vers.c -o build/temp.linux-aarch64-cpython-311/scratch/vers.o
/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang build/temp.linux-aarch64-cpython-311/scratch/vers.o -L/home/rudra/Documents/libzmq-builds/arm64/lib -Wl,--enable-new-dtags,-R/home/rudra/Documents/libzmq-builds/arm64/lib -lzmq -o build/temp.linux-aarch64-cpython-311/scratch/vers
Error running version detection script:

Failed to run ['build/temp.linux-aarch64-cpython-311/scratch/vers']: OSError(8, 'Exec format error')

error: Error running version detection script:

Failed to run ['build/temp.linux-aarch64-cpython-311/scratch/vers']: OSError(8, 'Exec format error')

Fatal: Falling back on bundled libzmq, but config has explicitly prohibited building the libzmq extension.

Code to reproduce bug

No response

Traceback, if applicable

No response

More info

No response

The text was updated successfully, but these errors were encountered:

Soft-Buddy · 2024-02-28T17:19:38Z

If I don't define prefix, it fails with :

In file included from bundled/zeromq/src/ipc_address.cpp:31:
bundled/zeromq/src/compat.hpp:45:1: error: static declaration of 'strlcpy' follows non-static declaration
strlcpy (char *dest_, const char *src_, const size_t dest_size_)
^
/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/string.h:167:8: note: previous declaration is here
size_t strlcpy(char* _Nonnull __dst, const char* _Nonnull __src, size_t __n);
       ^
1 error generated.
error: command '/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang++' failed with exit code 1

Soft-Buddy · 2024-02-28T17:39:40Z

If I somehow modify string.h from sysroot, it fails with :

bundled/zeromq/src/thread.cpp:371:11: error: use of undeclared identifier 'pthread_setaffinity_np'; did you mean 'sched_setaffinity'?
pthread_setaffinity_np (pthread_self (), sizeof (cpu_set_t), &cpuset);
^~~~~~~~~~~~~~~~~~~~~~
sched_setaffinity
/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/sched.h:243:5: note: 'sched_setaffinity' declared here
int sched_setaffinity(pid_t __pid, size_t __set_size, const cpu_set_t* _Nonnull __set);
^
1 error generated.
error: command '/android-ndk-r26c/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang++' failed with exit code 1

minrk · 2024-02-29T14:52:17Z

Thanks for testing! Can you try building the latest pyzmq v26 prerelease? Builds are completely rewritten, and in particular, the step where libzmq detects if strlcopy is available is no longer skipped.

Soft-Buddy · 2024-02-29T15:46:15Z

Thanks for testing! Can you try building the latest pyzmq v26 prerelease? Builds are completely rewritten, and in particular, the step where libzmq detects if strlcopy is available is no longer skipped.

Where can I find the pre-release ?
PS: Nvm I got it

Soft-Buddy · 2024-02-29T15:58:04Z

      *** Building project with Ninja...
      [1/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/precompiled.cpp.o
      [2/109] Building C object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/external/sha1/sha1.c.o
      [3/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/clock.cpp.o
      [4/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/channel.cpp.o
      [5/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/client.cpp.o
      [6/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/curve_mechanism_base.cpp.o
      [7/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/address.cpp.o
      [8/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/devpoll.cpp.o
      [9/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/dealer.cpp.o
      [10/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/curve_client.cpp.o
      [11/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/err.cpp.o
      [12/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/endpoint.cpp.o
      [13/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/dgram.cpp.o
      [14/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/curve_server.cpp.o
      [15/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/epoll.cpp.o
      [16/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/fq.cpp.o
      [17/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/io_thread.cpp.o
      [18/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/ip.cpp.o
      [19/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/io_object.cpp.o
      [20/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/dist.cpp.o
      [21/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/ipc_address.cpp.o
      [22/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/kqueue.cpp.o
      [23/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/ipc_connecter.cpp.o
      [24/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/lb.cpp.o
      [25/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/mailbox_safe.cpp.o
      [26/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/mailbox.cpp.o
      [27/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/norm_engine.cpp.o
      [28/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/mechanism_base.cpp.o
      [29/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/metadata.cpp.o
      [30/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/msg.cpp.o
      [31/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/ipc_listener.cpp.o
      [32/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/mtrie.cpp.o
      [33/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/mechanism.cpp.o
      [34/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pgm_receiver.cpp.o
      [35/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pgm_sender.cpp.o
      [36/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/null_mechanism.cpp.o
      [37/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pair.cpp.o
      [38/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pgm_socket.cpp.o
      [39/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/options.cpp.o
      [40/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/ctx.cpp.o
      [41/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/object.cpp.o
      [42/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/own.cpp.o
      [43/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/peer.cpp.o
      [44/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/polling_util.cpp.o
      [45/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/poller_base.cpp.o
      [46/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/poll.cpp.o
      [47/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/plain_client.cpp.o
      [48/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/plain_server.cpp.o
      [49/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/random.cpp.o
      [50/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pollset.cpp.o
      [51/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pub.cpp.o
      [52/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/raw_encoder.cpp.o
      [53/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/push.cpp.o
      [54/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pipe.cpp.o
      [55/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/pull.cpp.o
      [56/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/proxy.cpp.o
      [57/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/select.cpp.o
      [58/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/reaper.cpp.o
      [59/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/rep.cpp.o
      [60/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/raw_decoder.cpp.o
      [61/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/req.cpp.o
      [62/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/signaler.cpp.o
      [63/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/router.cpp.o
      [64/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/server.cpp.o
      [65/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/raw_engine.cpp.o
      [66/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/socks.cpp.o
      [67/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/socks_connecter.cpp.o
      [68/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/session_base.cpp.o
      [69/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tcp.cpp.o
      [70/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/sub.cpp.o
      [71/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tcp_address.cpp.o
      [72/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/stream.cpp.o
      [73/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tcp_connecter.cpp.o
      [74/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tcp_listener.cpp.o
      [75/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/trie.cpp.o
      [76/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/thread.cpp.o
      [77/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/v1_decoder.cpp.o
      [78/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/v1_encoder.cpp.o
      [79/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/radix_tree.cpp.o
      [80/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/v2_encoder.cpp.o
      [81/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/v3_1_encoder.cpp.o
      [82/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/stream_engine_base.cpp.o
      [83/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/zmq_utils.cpp.o
      [84/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/xsub.cpp.o
      [85/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/v2_decoder.cpp.o
      [86/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/decoder_allocators.cpp.o
      [87/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/timers.cpp.o
      [88/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/socket_base.cpp.o
      [89/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/zmq.cpp.o
      [90/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/socket_poller.cpp.o
      [91/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/udp_address.cpp.o
      [92/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/xpub.cpp.o
      [93/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/radio.cpp.o
      [94/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/gather.cpp.o
      [95/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/scatter.cpp.o
      [96/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/dish.cpp.o
      [97/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/ip_resolver.cpp.o
      [98/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/udp_engine.cpp.o
      [99/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tipc_address.cpp.o
      [100/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/zap_client.cpp.o
      [101/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/stream_connecter_base.cpp.o
      [102/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/stream_listener_base.cpp.o
      [103/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tipc_connecter.cpp.o
      [104/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/zmtp_engine.cpp.o
      [105/109] Building CXX object _deps/bundled_libzmq-build/CMakeFiles/objects.dir/src/tipc_listener.cpp.o
      [106/109] Linking CXX static library _deps/bundled_libzmq-build/lib/libzmq.a
      [107/109] Generating _src/_zmq.c
      FAILED: _src/_zmq.c /tmp/tmpubily8i6/build/_src/_zmq.c
      cd /tmp/tmpubily8i6/build && /usr/bin/cython --3str --output-file /tmp/tmpubily8i6/build/_src/_zmq.c --module-name zmq.backend.cython._zmq /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      
      # message Frame class
      
      _zhint = C.struct(
          sock=p_void,
          mutex=pointer(mutex_t),
                       ^
      ------------------------------------------------------------
      
      /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py:170:18: Unknown type
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      
      # message Frame class
      
      _zhint = C.struct(
          sock=p_void,
          mutex=pointer(mutex_t),
                      ^
      ------------------------------------------------------------
      
      /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py:170:17: Unknown type
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
              return 0
      
      
      @cfunc
      @inline
      def _copy_zmq_msg_bytes(zmq_msg: pointer(zmq_msg_t)) -> bytes:
      ^
      ------------------------------------------------------------
      
      /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py:222:0: Exception clause not allowed for function returning Python object
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
              """The address of the underlying libzmq context"""
              return cast(size_t, self.handle)
      
          @cfunc
          @inline
          def _term(self) -> C.int:
         ^
      ------------------------------------------------------------
      
      /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py:530:4: Signature not compatible with previous declaration
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
          cdef void *handle  # The C handle for the underlying zmq object.
          cdef bint _shadow  # whether the Context is a shadow wrapper of another
          cdef int _pid  # the pid of the process which created me (for fork safety)
      
          cdef public bint closed  # bool property for a closed context.
          cdef inline int _term(self)
                              ^
      ------------------------------------------------------------
      
      /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.pxd:17:25: Previous declaration is here
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
              if not zmq.has('draft'):
                  raise RuntimeError("libzmq must be built with draft support")
              rc: C.int = zmq_leave(self.handle, group)
              _check_rc(rc)
      
          def send(self, data, flags=0, copy: bint = True, track: bint = False):
                                       ^
      ------------------------------------------------------------
      
      /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py:1076:34: Compiler crash in AnalyseDeclarationsTransform
      
      File 'ModuleNode.py', line 124, in analyse_declarations: ModuleNode(_zmq.py:1:0,
          doc = 'Cython backend for pyzmq',
          full_module_name = 'zmq.backend.cython._zmq')
      File 'Nodes.py', line 431, in analyse_declarations: StatListNode(_zmq.py:6:0)
      File 'Nodes.py', line 431, in analyse_declarations: StatListNode(_zmq.py:622:0)
      File 'Nodes.py', line 375, in analyse_declarations: CompilerDirectivesNode(_zmq.py:622:0)
      File 'Nodes.py', line 431, in analyse_declarations: StatListNode(_zmq.py:622:0)
      File 'Nodes.py', line 4817, in analyse_declarations: CClassDefNode(_zmq.py:622:0,
          class_name = 'Socket',
          decorators = [...]/0,
          doc = '\n    A 0MQ socket.\n\n    These objects will generally be constructed via the socket() method of a Context object.\n\n    Note: 0MQ Sockets are *not* threadsafe. **DO NOT** share them across threads.\n\n    Parameters\n    ----------\n    context : Context\n        The 0MQ Context this Socket belongs to.\n    socket_type : int\n        The socket type, which can be any of the 0MQ socket types:\n        REQ, REP, PUB, SUB, PAIR, DEALER, ROUTER, PULL, PUSH, XPUB, XSUB.\n\n    See Also\n    --------\n    .Context.socket : method for creating a socket bound to a Context.\n    ',
          visibility = 'private')
      File 'Nodes.py', line 431, in analyse_declarations: StatListNode(_zmq.py:623:4)
      File 'Nodes.py', line 2359, in analyse_declarations: CFuncDefNode(_zmq.py:1076:4,
          doc = '\n        Send a single zmq message frame on this socket.\n\n        This queues the message to be sent by the IO thread at a later time.\n\n        With flags=NOBLOCK, this raises :class:`ZMQError` if the queue is full;\n        otherwise, this waits until space is available.\n        See :class:`Poller` for more general non-blocking I/O.\n\n        Parameters\n        ----------\n        data : bytes, Frame, memoryview\n            The content of the message. This can be any object that provides\n            the Python buffer API (`memoryview(data)` can be called).\n        flags : int\n            0, NOBLOCK, SNDMORE, or NOBLOCK|SNDMORE.\n        copy : bool\n            Should the message be sent in a copying or non-copying manner.\n        track : bool\n            Should the message be tracked for notification that ZMQ has\n            finished with it? (ignored if copy=True)\n\n        Returns\n        -------\n        None : if `copy` or not track\n            None if message was sent, raises an exception otherwise.\n        MessageTracker : if track and not copy\n            a MessageTracker object, whose `pending` property will\n            be True until the send is completed.\n\n        Raises\n        ------\n        TypeError\n            If a unicode object is passed\n        ValueError\n            If `track=True`, but an untracked Frame is passed.\n        ZMQError\n            for any of the reasons zmq_msg_send might fail (including\n            if NOBLOCK is set and the outgoing queue is full).\n\n        ',
          is_c_class_method = 1,
          modifiers = [...]/0,
          overridable = 1,
          visibility = 'private')
      File 'Nodes.py', line 681, in analyse: CFuncDeclaratorNode(_zmq.py:1076:4,
          calling_convention = '',
          optional_arg_count = 1)
      File 'Nodes.py', line 909, in analyse: CArgDeclNode(_zmq.py:1076:34,
          is_generic = 1,
          outer_attrs = [...]/2)
      
      Compiler crash traceback from this point on:
        File "/usr/lib/python3/dist-packages/Cython/Compiler/Nodes.py", line 909, in analyse
          if self.annotation and env and env.directives['annotation_typing'] and self.base_type.name is None:
                                                                                ^^^^^^^^^^^^^^^^^^^
      AttributeError: 'CAnalysedBaseTypeNode' object has no attribute 'name'
      ninja: build stopped: subcommand failed.
      
      *** CMake build failed
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyzmq
Failed to build pyzmq
ERROR: Failed to build one or more wheels

It fails with this error buddy

minrk · 2024-03-01T09:07:06Z

cd /tmp/tmpubily8i6/build && /usr/bin/cython --3str --output-file /tmp/tmpubily8i6/build/_src/_zmq.c --module-name zmq.backend.cython._zmq /home/rudra/Documents/compilation/aarch64/pyzmq/zmq/backend/cython/_zmq.py

This is the problem, it's finding /usr/bin/cython instead of the one in the build env. I'll have a look to see if I can make that more reliable.

minrk · 2024-03-01T09:15:25Z

Can you share the command you are using to build?

It should be e.g. pip wheel --pre pyzmq, or if you have a local checkout, python3 -m build --wheel .

Also, what is /usr/bin/cython --version?

Soft-Buddy · 2024-03-01T19:05:21Z

I guess the pre-release is kindaaaaaaa weird buddy. It's building manylinux wheels for aarch64.

My cython version : Version: 3.0.8
The wheel being generated :
pyzmq-26.0.0b1-cp311-cp311-manylinux_2_38_aarch64.whl.zip
Android doesn't support manylinux builds.
The appropriate wheels packaged for android are like : <modulename>-<moduleversion>-cp311-cp311-linux_aarch64.whl but I instead get manulinux builds.

I Appreciate your assistance so far :)

Soft-Buddy · 2024-03-01T19:06:06Z

My build command is : pip wheel --pre .

minrk · 2024-03-01T22:18:07Z

I'm not sure what's responsible for the manylinux tag (it isn't pyzmq), but I'm also not 100% sure it's a problem. Does this mean you got a successful build now?

If it's just the tag, you can relabel wheels with:

 python3 -m wheel tags --platform-tag linux_aarch64 pyzmq-26.0.0b1-cp311-cp311-manylinux_2_38_aarch64.whl

but maybe it's indicative of not having picked up the right configuration.

I don't suppose you have setup scripts or a docker image you use to do builds? It's hard for me to debug remotely, and I don't know how to setup a cross-compiling environment, except for on conda-forge where the crossenv builds are working fine.

Soft-Buddy · 2024-03-02T15:41:28Z

@minrk I setup cross compilation using python -m crossenv <path to the cross compiled python binary> <virtual env folder name>, If you need I can share the cross compiled python with you.

minrk · 2024-03-02T15:54:28Z

Yeah, I've used crossenv in conda-forge, but I don't know how to set up the environment in which to install crossenv (i.e. the toolchain, target python and host python, etc.). If you have a docker image or something to reproduce the environment, that would be super helpful.

Soft-Buddy · 2024-03-02T16:03:04Z

The sad part is that I use it on my pc directly (cause I need to perform modifications for each architecture sometimes😅), I can share tar of the host python and the instructions at the moment.

minrk · 2024-03-03T18:41:12Z

Anything you can provide will help. Even links to the setup instructions you use.

minrk · 2024-03-06T15:37:29Z

I still don't know how to setup (or test) a build for android, but I have managed to build the latest pyzmq prerelease with crossenv. In my case, I cross-compiled from aarch64 to x86_64, and it built a wheel that I could then install and use in an x86_64 container (python:3.11).

I used this Dockerfile.

It seems with $CC and $CXX defined, it works. I either needed to build libsodium and libzmq with --host $HOST before linking from pyzmq, or build static libzmq & libsodium by specifying PYZMQ_LIBSODIUM_CONFIGURE_ARGS="--host $HOST".

I initially thought pyzmq should be able to do this automatically, but it doesn't seem to know that it's cross compiling even though it is, so it's hard to set the host triplet. Setting it directly works, though.

It's unclear how much can be expected to work for cross compiles without any help, but at least I now know for sure that building with crossenv does work. We just need to understand what's going on with your android env.

Soft-Buddy · 2024-03-06T15:41:43Z

https://drive.google.com/file/d/1wBnjQXRej9-E9iY542RhdJfDDxglYKNa/view?usp=sharing

Here's my build. To use it, you'll need android ndk r26c download.

Any help is very very much appreciated

minrk · 2024-03-11T13:52:05Z

This dockerfile appears to successfully cross-compile pyzmq linked against libzmq and libsodium for android.

I have no way of testing the results, but let me know if that helps.

Soft-Buddy · 2024-03-11T13:54:09Z

I'll check it, I have my own app where I can test...

minrk · 2024-03-26T13:10:27Z

@Soft-Buddy have you had any chance to test? So far, everything I see looks like it works, but I have no environment to test the results with.

minrk · 2024-04-15T08:36:22Z

Closing this, since I have a working documented example. Feel free to open a new issue if you find one.

minrk mentioned this issue Mar 15, 2024

add cross-compile examples to docs #1964

Merged

minrk closed this as completed Apr 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Error compiling wheels using crossenv #1957

BUG: Error compiling wheels using crossenv #1957

Soft-Buddy commented Feb 28, 2024

Soft-Buddy commented Feb 28, 2024

Soft-Buddy commented Feb 28, 2024

minrk commented Feb 29, 2024 •

edited

Loading

Soft-Buddy commented Feb 29, 2024 •

edited

Loading

Soft-Buddy commented Feb 29, 2024

minrk commented Mar 1, 2024

minrk commented Mar 1, 2024

Soft-Buddy commented Mar 1, 2024

Soft-Buddy commented Mar 1, 2024

minrk commented Mar 1, 2024

Soft-Buddy commented Mar 2, 2024

minrk commented Mar 2, 2024

Soft-Buddy commented Mar 2, 2024

minrk commented Mar 3, 2024

minrk commented Mar 6, 2024

Soft-Buddy commented Mar 6, 2024

minrk commented Mar 11, 2024

Soft-Buddy commented Mar 11, 2024

minrk commented Mar 26, 2024

minrk commented Apr 15, 2024

BUG: Error compiling wheels using crossenv #1957

BUG: Error compiling wheels using crossenv #1957

Comments

Soft-Buddy commented Feb 28, 2024

This is a pyzmq bug

What pyzmq version?

What libzmq version?

Python version (and how it was installed)

OS

What happened?

Code to reproduce bug

Traceback, if applicable

More info

Soft-Buddy commented Feb 28, 2024

Soft-Buddy commented Feb 28, 2024

minrk commented Feb 29, 2024 • edited Loading

Soft-Buddy commented Feb 29, 2024 • edited Loading

Soft-Buddy commented Feb 29, 2024

minrk commented Mar 1, 2024

minrk commented Mar 1, 2024

Soft-Buddy commented Mar 1, 2024

Soft-Buddy commented Mar 1, 2024

minrk commented Mar 1, 2024

Soft-Buddy commented Mar 2, 2024

minrk commented Mar 2, 2024

Soft-Buddy commented Mar 2, 2024

minrk commented Mar 3, 2024

minrk commented Mar 6, 2024

Soft-Buddy commented Mar 6, 2024

minrk commented Mar 11, 2024

Soft-Buddy commented Mar 11, 2024

minrk commented Mar 26, 2024

minrk commented Apr 15, 2024

minrk commented Feb 29, 2024 •

edited

Loading

Soft-Buddy commented Feb 29, 2024 •

edited

Loading