Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

numpy recipe bug arm64-v8a #1873

Closed
musarra opened this issue Jun 18, 2019 · 39 comments · Fixed by #1892
Closed

numpy recipe bug arm64-v8a #1873

musarra opened this issue Jun 18, 2019 · 39 comments · Fixed by #1892

Comments

@musarra
Copy link

musarra commented Jun 18, 2019 •
edited by inclement
Loading

Versions

  • Python: 3
  • OS: linux
  • Kivy: 1.11
  • Cython: not sure

Description

// REPLACE ME: What are you trying to get done, what has happened, what went wrong, and what did you expect?

Google Play does not support 32 bit libraries and the armeabi-v7a cannot be used. This means the apk I created and worked for months could not be uploaded. I tried to fix that issue changing the architecture to 64 bit arm64-v8a. Problem is that I have numpy in requirements and it has an issue with the recipe. The apk get's built but crashes immediately, it never opens.

I am not expert programmer but my suggestions to solve/check the issue looking to numpy recipe are:

  1. this url = 'https://pypi.python.org/packages/source.....etc' does not exist. I would replace for https://pypi.org/simple/numpy/numpy-1.16.4.zip instead
  2. in recipe, version = '1.15.1'. In numpy website, version = 1.16.4

I built a very simple HelloWorld app in kivy to demonstrate this issue below.

Command:

buildozer android debug

Logs

06/18 23:34:21: Launching hwtest-0.1-debug
$ adb shell am start -n "org.test.hwtest/org.kivy.android.PythonActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER
Client not ready yet..Waiting for process to come online
Waiting for process to come online
Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page.
I/python: [INFO ] [Factory ] 184 symbols loaded
I/python: [INFO ] [Image ] Providers: img_tex, img_dds, img_sdl2, img_gif (img_pil, img_ffpyplayer ignored)
I/python: Traceback (most recent call last):
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/core/init.py", line 16, in
I/python: ImportError: dlopen failed: cannot locate symbol "__fixunstfsi" referenced by "/data/data/org.test.hwtest/files/app/_python_bundle/site-packages/numpy/core/multiarray.so"...

 During handling of the above exception, another exception occurred:
 
 Traceback (most recent call last):
   File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/app/main.py", line 3, in <module>

I/python: File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/init.py", line 142, in
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/add_newdocs.py", line 13, in
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/lib/init.py", line 8, in
I/python: File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/lib/type_check.py", line 11, in
File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/python-installs/hwtest/numpy/core/init.py", line 26, in
ImportError:
I/python: Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.

 Original error was: dlopen failed: cannot locate symbol "__fixunstfsi" referenced by "/data/data/org.test.hwtest/files/app/_python_bundle/site-packages/numpy/core/multiarray.so"...
 
Python for android ended.

W/IPCThreadState: Calling IPCThreadState::self() during shutdown is dangerous, expect a crash.
W/org.test.hwtes: 0xebadde09 skipped times: 0
A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x184 in tid 26345 (org.test.hwtest), pid 26345 (org.test.hwtest)
Application terminated.
@RobertFlatt
Copy link
Contributor

Saw same issue with 64 bit (but not 32 bit) numpy build.

The numpy release notes https://github.com/numpy/numpy/releases contain the following under release 1.16.0

Highlight
Improved support for the ARM

and later

ARM support updated
Support for ARM CPUs has been updated to accommodate 32 and 64 bit targets,
and also big and little endian byte ordering. AARCH32 memory alignment issues
have been addressed. CI testing has been expanded to include AARCH64 targets
via the services of shippable.com.

This might be saying ARM 64 bit support added? Perhaps the issue is related to defaulting to 1.15.1 ?

Tried requirements numpy==1.16.0 but this gives a build (fix-numpy.patch) error.

@musarra
Copy link
Author

musarra commented Jun 19, 2019 via email

@inclement
Copy link
Member

@Ham-Merhead Thanks for the note, it looks like updating to numpy 1.16.0 is probably the obvious thing worth trying.

@RobertFlatt
Copy link
Contributor

@inclement
Been trying to build numpy 1.16.4 using the p4a-recipes directory
Issues with fix-numpy.patch , and python-fixes.patch , other patches are OK

@musarra
Copy link
Author

musarra commented Jun 20, 2019 via email

@inclement
Copy link
Member

@musarra You'll need to try what @Ham-Merhead mentioned, updating the patches as necessary for numpy 1.16. This isn't that hard, but may be tricky to get into if you aren't familiar with it.

I'll try to look at this at the weekend, I actually have a 64 bit phone now to test on.

@musarra
Copy link
Author

musarra commented Jun 20, 2019 via email

@musarra
Copy link
Author

musarra commented Jun 21, 2019 •
edited by inclement
Loading

By the way guys, I think is not worth it to open another bug tag, but in my simple HelloWorld android app, if I set py file to 'import matplotlib.pyplot as plt' and buildozer.spec to 'requirements = python3,kivy,matplotlib' it does not build the apk and the error message is below. Guess what? is related to numpy....

[INFO]:    Building matplotlib for arm64-v8a
[INFO]:    matplotlib apparently isn't already in site-packages
[INFO]:    Building compiled components in matplotlib
[INFO]:    -> directory context /home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/matplotlib-python3/arm64-v8a__ndk_target_21/matplotlib
[INFO]:    -> running python setup.py build_ext -v
           Exception in thread background thread for pid 9686:                 
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 2459, in background_thread
    handle_exit_code(exit_code)
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 2157, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 815, in handle_command_exit_code
    raise exc
ErrorReturnCode_1: 

  RAN: /home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python setup.py build_ext -v

  STDOUT:
============================================================================
Edit setup.cfg to change the build options

BUILDING MATPLOTLIB
            matplotlib: yes [3.0.3]
                python: yes [3.7.1 (default, Jun 21 2019, 19:47:23)  [GCC
                        7.4.0]]
              platform: yes [linux]

REQUIRED DEPENDENCIES AND EXTENSIONS
                 numpy: yes [not found. pip may install it below.]
      install_requires: yes [handled by setuptools]
                libagg: yes [pkg-config information for 'libagg' could not
                        be found. Using local copy.]
              freetype: yes [version 2.8.1]
                   png: yes [version 1.6.34]
                 qhull: yes [pkg-config information for 'libqhull' could not
                        be found. Using local copy.]

OPTIONAL SUBPACKAGES
           sample_data: no  [skipping due to configuration]
              toolkits: yes [installing]
                 tests: no  [skipping due to configuration]
        toolkits_tests: no  [skipping due to configuration]

OPTIONAL BACKEND EXTENSIONS
                   agg: yes [installing]
                 tkagg: no  [Disabled by patching during Android build]
                macosx: no  [skipping due to configuration]
             windowing: no  [skipping due to configuration]

OPTIONAL PACKAGE DATA
                  dlls: no  [skipping due to configuration]

**Download error on https://pypi.org/simple/numpy/: unknown url type: https -- Some packages may not be found!**
**Couldn't find index page for 'numpy' (maybe misspelled?)**
**Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!**
**No local packages or working download links found for numpy==1.15.1**
Traceback (most recent call last):
  File "setup.py", line 297, in <module>
    cmdclass=cmdclass,
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 144, in setup
    _install_setup_requires(attrs)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
    return cmd.easy_install(req)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('numpy==1.15.1')


  STDERR:


Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 1186, in <module>
    main()
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 1180, in main
    ToolchainCL()
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 663, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 152, in wrapper_func
    build_dist_from_args(ctx, dist, args)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/python-for-android/pythonforandroid/toolchain.py", line 205, in build_dist_from_args
    args, "ignore_setup_py", False
  File "pythonforandroid/build.py", line 555, in build_recipes
  File "pythonforandroid/recipe.py", line 883, in build_arch
  File "pythonforandroid/recipe.py", line 938, in build_compiled_components
  File "pythonforandroid/recipe.py", line 895, in build_compiled_components
  File "pythonforandroid/logger.py", line 178, in shprint
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 863, in next
    self.wait()
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/local/lib/python2.7/dist-packages/sh-1.12.14-py2.7.egg/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python setup.py build_ext -v

  STDOUT:
============================================================================
Edit setup.cfg to change the build options

BUILDING MATPLOTLIB
            matplotlib: yes [3.0.3]
                python: yes [3.7.1 (default, Jun 21 2019, 19:47:23)  [GCC
                        7.4.0]]
              platform: yes [linux]

REQUIRED DEPENDENCIES AND EXTENSIONS
                 numpy: yes [not found. pip may install it below.]
      install_requires: yes [handled by setuptools]
                libagg: yes [pkg-config information for 'libagg' could not
                        be found. Using local copy.]
              freetype: yes [version 2.8.1]
                   png: yes [version 1.6.34]
                 qhull: yes [pkg-config information for 'libqhull' could not
                        be found. Using local copy.]

OPTIONAL SUBPACKAGES
           sample_data: no  [skipping due to configuration]
              toolkits: yes [installing]
                 tests: no  [skipping due to configuration]
        toolkits_tests: no  [skipping due to configuration]

OPTIONAL BACKEND EXTENSIONS
                   agg: yes [installing]
                 tkagg: no  [Disabled by patching during Android build]
                macosx: no  [skipping due to configuration]
             windowing: no  [skipping due to configuration]

OPTIONAL PACKAGE DATA
                  dlls: no  [skipping due to configuration]

Download error on https://pypi.org/simple/numpy/: unknown url type: https -- Some packages may not be found!
Couldn't find index page for 'numpy' (maybe misspelled?)
Download error on https://pypi.org/simple/: unknown url type: https -- Some packages may not be found!
No local packages or working download links found for numpy==1.15.1
Traceback (most recent call last):
  File "setup.py", line 297, in <module>
    cmdclass=cmdclass,
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 144, in setup
    _install_setup_requires(attrs)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 782, in resolve
    replace_conflicting=replace_conflicting
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1065, in best_match
    return self.obtain(req, installer)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/pkg_resources/__init__.py", line 1077, in obtain
    return installer(requirement)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/dist.py", line 791, in fetch_build_egg
    return cmd.easy_install(req)
  File "/home/musarra/Documents/pyprojects/helloworld/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/setuptools/command/easy_install.py", line 673, in easy_install
    raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('numpy==1.15.1')

@RobertFlatt
Copy link
Contributor

Some progress, but still stuff to figure out:

For 1.16.4 in the patches list I

  1. commented fix-numpy.patch -- because I have no understanding of the test methodology
  2. built a new python-fixes.patch (attached) -- no judgement here, just doing the same thing as before
    python-fixes.zip

However the build fails
error: Command "aarch64-linux-android-ld ............... failed with exit status 127

Don't know why aarch64-linux-android-ld is not found, except that I cant find it in python-for-android/build .

@musarra This works for p4a , Buildozer I don't know about:

  1. Create a directory "p4a-recipes" in your project directory
  2. copy p4a numpy recipe there from github
  3. replace python-fixes.patch with the one attached
  4. Edit __init__.py
    a) change version
    b) comment line containing 'fix-numpy.patch'
    c) change 'patches' in the following 4 lines to an absolute path e.g. '/usr/me/stuff/patches' (or whatever)

p4a will build numpy from these local instructions in place of the default instructions. (To check it is really building from there, insert a typo so the build fails!)

See if you can make some progress....

@musarra
Copy link
Author

musarra commented Jun 22, 2019 via email

@opacam opacam mentioned this issue Jun 22, 2019
Merged
7 tasks
@musarra
Copy link
Author

musarra commented Jun 24, 2019 via email

@inclement
Copy link
Member

@opacam Mentioned that it worked in a 64 bit build with ndk 19, in #1722

@musarra
Copy link
Author

musarra commented Jun 24, 2019 via email

@RobertFlatt
Copy link
Contributor

@musarra
I don't think Buildozer supports 19 yet (read the comments further up #1722 ).
I tried 19c with p4a (64 bit target), I get this message:

[INFO]:    Found NDK revision 19.2.5345600
[WARNING]: Maximum recommended NDK version is 17c
[WARNING]: Newer NDKs may not be fully supported by p4a.

And after a while it fails (as expected):
[ERROR]: Build failed: Couldn't find executable for CC. This indicates a problem locating the aarch64-linux-android-gcc executable in the Android NDK, not that you don't have a normal compiler installed. Exiting.

aarch64-linux-android-gcc is not part of 19c , and p4a/Bulldozer tool flow changes (in this case to clang) are required for 19. So I would say for the moment we cant get there from here (via NDK 19). However NDK 19 looks like it will be a future route.

@musarra
Copy link
Author

musarra commented Jun 26, 2019 via email

@inclement
Copy link
Member

I've been experimenting with a numpy 1.16.4 recipe, but fighting some weird linker errors at the moment. I also don't know if this will fix the problem.

@musarra
Copy link
Author

musarra commented Jun 26, 2019 via email

@musarra
Copy link
Author

musarra commented Jun 26, 2019 via email

@inclement
Copy link
Member

There's some doc at https://python-for-android.readthedocs.io/en/latest/recipes/, but mostly the existing recipes and code are the documentation. I've been meaning to write something about it, but for something like numpy you'll tend to have to work through compilation issues that aren't easily documented so there's a limit to what a tutorial can do.

@musarra
Copy link
Author

musarra commented Jun 26, 2019 via email

@inclement inclement mentioned this issue Jun 26, 2019
Merged
@inclement
Copy link
Member

inclement commented Jun 26, 2019 •
edited
Loading

I got 1.16.4 working and seems to run fine under armeabi-v7a and arm64-v8a. PR at #1892 for reference but it's in an untidy half-finished state right now. I'll finish it within the next few days.

@musarra
Copy link
Author

musarra commented Jun 26, 2019 via email

@musarra
Copy link
Author

musarra commented Jun 26, 2019 via email

@musarra
Copy link
Author

musarra commented Jun 29, 2019 via email

@inclement
Copy link
Member

@musarra This won't be in the release branch that buildozer uses for a while, you need to use p4a master or set the p4a.branch in buildozer.spec to develop. If you already did that, please post the log.

@musarra
Copy link
Author

musarra commented Jun 29, 2019 via email

@inclement
Copy link
Member

The matplotlib issue is probably environment related somehow, openssl is probably missing in the local python build. I'm not sure about the details though, I don't know what's different about my local env (which works okay). There's another issue open to track this.

@musarra
Copy link
Author

musarra commented Jun 29, 2019 via email

@AndreMiras
Copy link
Member

No it's definitely what @inclement said. The system doesn't have socket or other lib build with ssl support hence https schema not supported

@musarra
Copy link
Author

musarra commented Jun 29, 2019 via email

@AndreMiras
Copy link
Member

I would suggest to make sure your system libs are not broken https://stackoverflow.com/questions/28376506/urllib-https-request-urlopen-error-unknown-url-type-https
This comes outside p4a to me. By the way it seems like your run p4a from python2, plus some of the deps (e.g. sh) were installed via sudo pip install which I would not recommend. I would recommend using pip install --user instead, that way it's easy to wipe out everything if you ever mess with your deps

@musarra
Copy link
Author

musarra commented Jun 29, 2019 via email

@musarra
Copy link
Author

musarra commented Jun 30, 2019 via email

@AndreMiras
Copy link
Member

Glad you could solve it only installing some system deps.
As for how to use matplotlib, I've no clue, but you should check that from their website or tutorial on the web.
Let's migrate support conversation to Discord then. Good luck!

@musarra
Copy link
Author

musarra commented Jun 30, 2019 via email

@AndreMiras
Copy link
Member

I see that makes sense. Well I've never played with it, but I'm sure you should find someone on Discord.

@AndreMiras AndreMiras mentioned this issue Nov 5, 2019
Closed
@Fabiocke
Copy link

I solve the problem and it's very simple, just add: numpy == 1.16.4
in your buildozer.spec requiriments

@inclement
Copy link
Member

numpy 1.16.4 is the current default, as of a few months ago.

@GaspHard
Copy link

This does not work for me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Updated numpy recipe to version 1.16.4 inclement/python-for-android
6 participants
@AndreMiras @inclement @GaspHard @RobertFlatt @musarra @Fabiocke