bpo-30386: Add a build infrastructure for Android. #1629

xdegaye · 2017-05-17T14:17:06Z

Add a build infrastructure for Android to cross-compile Python using the Android NDK and allowing to run python interactively on the emulator or to run the Python test suite using the Android SDK.

https://bugs.python.org/issue30386

AraHaan · 2017-05-17T14:35:18Z

Mind also doing this for iOS as well so people can build their own free version of python for the iOS. It is annoying having to pay $$$ for 3.4+ on it but nothing for 2.7...

Also what about pip on the android and iOS? I thought they might not get pip to work on them.

xdegaye · 2017-05-30T18:33:12Z

Commit details can be found at issue bpo-30386.

xdegaye · 2017-08-08T21:49:30Z

xdegaye requested a review from tiran as a code owner 5 hours ago

I did not request a review from tiran.

…es from the external libraries

terminfo/l/linux is now again installed on the emulator.

See libffi/libffi#265. The external libraries are now untared in the build tree instead of the source tree previously.

This is at Android build-tools version 25.0.3 and emulator version 26.0.3.

pmp-p · 2017-11-17T18:05:57Z

@xdegaye is there some dedicated place to discuss about android ports since you are not on irc ?

xdegaye · 2017-11-17T20:49:24Z

@pmp-p you can open a new issue on the issue tracker and if you want to involve me in the discussion just add xdegaye to the nosy list of this issue, or you can send an email to python-ideas or python-dev.
It depends of what you want to discuss.

vstinner · 2017-12-12T13:29:57Z

Android/README.rst

+.. contents::
+
+This document provides a quick overview of the Python build system for Android
+on a **linux platform** [1]_.


This documentation is uncomplete. You need "quick start" instructions to explain what is Android, what do I require on my local host, do I need a physical device, etc. This documentation mostly explains the implementation, not really how to use it.

I'm trying to run Python on Android in an emulator using these recipes, but it's hard to understand how to use it. First, I downloaded NDK r16b but configure-android immediately fails because it requires NDK 14... This thing be very explicit in the doc, since the NDK is a big file (800 MB, so slow to download) and takes a lot of disk space (2.6 GB). It's not easy to install it.

I will add a quick start section shortly.

First, I downloaded NDK r16b but configure-android immediately fails because it requires NDK 14.

I understand your frustation. The README does say that android-ndk-r14 is required in a note and that note is referred to in each Requirements section where the NDK is required. I will add a warning about this in the quick start section unless you have a better idea.

bedevere-bot · 2017-12-12T13:30:22Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

vstinner · 2017-12-12T13:55:20Z

I did not request a review from tiran.

See .github/CODEOWNERS: some maintainers are automatically added depending which files are modified.

vstinner · 2017-12-12T15:10:45Z

I tried to use the recipe but I failed. There many traps:

NDK 14 is required, whereas Android website first proposes NDK 16
NDK 14 requires ncurses5 whereas Fedora 27 uses ncurses6: "dnf install ncurses-compat-libs" is needed on Fedora 27
Python 3.7 must be installed on the system, available in the PATH

I tried:

cd Android
../configure-android  PYTHON_FOR_BUILD=/opt/py37/bin/python3.7
make

The compilation fails somewhere in setup.py about Tools/. I commented Tools in setup.py, then the compilation fails on building something about extensions:

vstinner@apu$ make
running build
running build_ext
Traceback (most recent call last):
  File "/home/vstinner/prog/python/master/setup.py", line 2310, in <module>
    main()
  File "/home/vstinner/prog/python/master/setup.py", line 2299, in main
    ext_modules=[Extension('_struct', ['_struct.c'])]
  File "/opt/py37/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/py37/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/opt/py37/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/py37/lib/python3.7/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/opt/py37/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/opt/py37/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/opt/py37/lib/python3.7/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/home/vstinner/prog/python/master/setup.py", line 251, in build_extensions
    for filename in self.distribution.scripts]
TypeError: 'NoneType' object is not iterable

vstinner · 2017-12-12T15:17:45Z


  File "/home/vstinner/prog/python/master/setup.py", line 251, in build_extensions
    for filename in self.distribution.scripts]
TypeError: 'NoneType' object is not iterable

Ah! This is caused by my setup.py hack. The correct patch is:

diff --git a/setup.py b/setup.py
index 09e35e5060..965d7d58fd 100644
--- a/setup.py
+++ b/setup.py
@@ -2301,8 +2301,7 @@ def main():
           # If you change the scripts installed here, you also need to
           # check the PyBuildScripts command above, and change the links
           # created by the bininstall target in Makefile.pre.in
-          scripts = ["Tools/scripts/pydoc3", "Tools/scripts/idle3",
-                     "Tools/scripts/2to3", "Tools/scripts/pyvenv"]
+          scripts = []
         )
 
 # --install-platlib

With this patch, I'm able to build Python:

Python build finished successfully!

Ok. Now what I am supposed to do? :-)

vstinner · 2017-12-12T15:48:15Z

My notes:

Quick Start
===========

Main steps:

* Download NDK and Android x86 VM ISO
* Compile Python 3.7 on the host
* Install NDK
* Cross-Compile Python to Android
* Create an Android VM
* Copy the ELF ./python binary to the VM
* Start the VM, run ./python

Requirements:

* Fedora 27
* qemu
* 3 GB disk space at least: NDK (2.6 GB) + 400 MB of Python build objects + 700 MB ISO file

Download:

* Dowload NDK 14 from https://developer.android.com/ndk/downloads/index.html -- NDK 14 is an old version, don't pick the most recent!
* Download android-x86_64-6.0-r3.iso from http://www.android-x86.org/download

Instructions.

* NDK: toolchains + platforms, 800 MiB download, 2.6 GB uncompressed
* Agree with the Android Terms and Conditions
* Download android-ndk-r14: Apache license, 813 MB


Compile Python 3.7 on the host::

  ./configure --prefix /opt/py37
  make
  make install

Install NDK::

  mkdir -p /opt/android
  unzip android-ndk-r16b-linux-x86_64.zip -d /opt/android
  export ANDROID_NDK_ROOT=/opt/android/android-ndk-r14b

  sudo dnf install ncurses-compat-libs # Fedora
  # ArchLinux: https://aur.archlinux.org/packages/android-ndk/

Fix setup.py::

  diff --git a/setup.py b/setup.py
  index 09e35e5060..965d7d58fd 100644
  --- a/setup.py
  +++ b/setup.py
  @@ -2301,8 +2301,7 @@ def main():
             # If you change the scripts installed here, you also need to
             # check the PyBuildScripts command above, and change the links
             # created by the bininstall target in Makefile.pre.in
  -          scripts = ["Tools/scripts/pydoc3", "Tools/scripts/idle3",
  -                     "Tools/scripts/2to3", "Tools/scripts/pyvenv"]
  +          scripts = []
           )

   # --install-platlib

Cross-compile::

  cd Android
  ../configure-android --with-pydebug PYTHON_FOR_BUILD=/opt/py37/bin/python3.7
  make

pmp-p · 2017-12-12T16:32:26Z

Hi, first thx for layout some android support.

linux mint 18.3 : ANDROID_ARCH=armv7 ANDROID_API=21
ANDROID_NDK_ROOT set to ndk 14b

restrictions of WITH_LIBRARIES="libffi" cause make install to fail because no .so get generated and build.mk tries to copy and package them ( libffi is static .a scripts looks for .so ).

btw curses build fails with : ../ncurses/./tty/hardscroll.c:148:10: fatal error: 'curses.priv.h' file not found
side note android libs are not numbered libpython :
patchelf --print-soname libpython3.7m.so => libpython3.7m.so.1.0 shoud be libpython3.7m.so

xdegaye · 2017-12-12T17:05:49Z

@vstinner

Ok. Now what I am supposed to do? :-)

Sorry but you did it wrong :-(. To build Python for an emulator you must follow the Build for the emulator with makesetup section of the README. Do not run configure-android directly, this script is run by the Makefile created by makesetup.

So you follow these steps:

Install the NDK.
Install the SDK (check the requirements for disk space). The SDK contains qemu so no need to install it and it contains the images for the architectures that you are downloading with the sdkmanager command so the Android x86 VM ISO image is not needed.
Run makesetup to create the Makefile and refer to the section Makefile targets for the description of the Makefile targets.
Run make build. This target compiles Python 3.7 natively (it is not necessary to do it yourself) and cross-compiles the external libraries and Python.
Run make install. This target makes a distribution, creates the AVD, starts the emulator, installs python and runs a remote shell. Creation of the AVD takes quite a long time and is done only once.

I think the Cross-compilation with configure-android section should be removed, it is confusing.

xdegaye · 2017-12-12T17:19:52Z

@pmp-p Are you using the external libraries that are downloaded by the Makefile generated by makesetup and are they built by this Makefile ?
In the case where the answer is yes to both questions, can you please attach the full output of the build.

pmp-p · 2017-12-12T18:24:59Z

yes & yes : here's full session, https://gist.githubusercontent.com/pmp-p/84e93c3e13943ea61e5a3493debf23de/raw/5696180869ccee445c47af9b07d0f3c67d8f6d6d/gistfile1.txt

and if it can help to broad support, i tried cross compilation too for api-19 at the time of 3.7a2 i got repl, interpreter to embed /run panda3d both on H3Droid armv7 : here are some ugly patches https://github.com/pmp-p/h3droid/tree/sdk/usr/src/python3.7-android/patches

xdegaye · 2017-12-13T09:20:26Z

@pmp-p

yes & yes

Are you sure ?

The ncurses compilation recipe that fails in your build starts with cd ../obj_s; and there is no such statement anywhere in the Makefile generated from the ncurses sources downloaded by this build system (and of course the cd ../obj_s; statement does not show up when reproducing the build with your command line arguments). And BTW thanks for having me lose my time searching for this.
It is not possible that the problem that you have reported in your previous post about libffi may occur unless this build system has been modified.

pmp-p · 2017-12-13T19:17:04Z

( "BTW thanks for having me lose my time searching for this." Then please for god's sake don't make android support official if this is the kind of help or welcome we ( poor users) would get. )

Of course i am sure , i took the time to build a clean mint 18 pc to run the build side by side with victor he was building api-24 64bits, while me api-21 32bits.

Parenthèses fermées j'espère.

PS: by reviewing irc log, it appears Victor had it built for intel/amd, not arm like me maybe that's a clue
"It is not possible" => impossible n'est pas français donc si ;)

xdegaye · 2017-12-14T07:56:06Z

This is the full build.log of a build that uses the same parameters as those used by @pmp-p. The differences with @pmp-p results are:

The build is SUCCESSFUL.
libffi source is downloaded and built.
readline source is downloaded (which is not the case with the build output reported by @pmp-p, the origin of the readline source is unknown) and successfully built.

BTW still missing the build output of the other problem reported @pmp-p.

pmp-p · 2017-12-14T18:17:41Z

"the origin of the readline source is unknown" it is always the case since you are using multiple external link .tar.gz without checking checksums. In some case that could be a concern at least for some devices and average system administrator.

You are using ccache and may have curses-dev installed so i did the same, checked both using mawk/gawk => but problem was still there.

For me it is quite simple : ncurses cross compile support is not as good as expected and need controlled environnement or a script of yours to help it.

Sadly i tried on ubuntu flavor because that is what less than average unix developper is using nowadays.
i suggest you set only libffi by default and sanitize environnement before building deps.

i'll post make install log failure when make depends step is fixed, as i adopted your script for a panda3d android build process : i'll try to investigate more.

pmp-p · 2018-01-21T14:55:15Z

Well i upgraded my system recently, and the ncurses build problem is gone, so let's forget about it.

But some questions arose ...

CPython compiles fines with any ndk : only third parties have real trouble, so why introduce them ?

Are you removing older api and other ndk just because you know some tests won't run ?

As running/embedding/testing is a different problem and is not limited to "api-21 ndk-14 on friendly host" now i get it : that's not android platform support of any kind so i owe you an apology because i was confused about that ( maybe false hope ),

I suggest you rename the title to "add a build infrastucture for 1 specific Android version where all third parties are not completely broken" and i wanted to thank you for your work it was very helpfull for a specific use case.

sys.prefix must not be located on /sdcard: pip adds a sys.prefix/bin/ directory and fails upon PermissionError raised when using the /sdcard file system.

inclement · 2018-09-18T21:03:35Z

I just tested this PR for use with Kivy via python-for-android, something I've been meaning to do for a long time, and it worked very well. I was also able to get target API 27 with the latest NDK 17 working without too many issues, although I don't really understand why some things did seem to break.

I found the python-dev mailing list posts about this but I'm not clear what is the current status of these efforts. Is this still the most up to date and active CPython for Android resource, or is there anything more recent, or likely to come soon? This PR already makes a superb base for improving python-for-android's Python 3 support, but I'd like to try to engage more with current developments if possible.

xdegaye · 2018-09-19T07:33:18Z

@inclement
I have made some changes to this PR few months ago that address Victor's review but did not push them here. I am not a core developer anymore and Victor has stated in two different issues that he is not interested anymore in Android, so it seems that it would be better to ask these questions on python-dev.

bpo-30386: Add a build infrastructure for Android.

65edf88

xdegaye added the type-feature A feature request or enhancement label May 17, 2017

xdegaye self-assigned this May 17, 2017

the-knights-who-say-ni added the CLA signed label May 17, 2017

bpo-30386: Support the new NDK Unified Headers.

67b40a7

ned-deily self-requested a review May 24, 2017 23:50

xdegaye added 3 commits May 27, 2017 21:43

Add Makefile targets to implement the buildbot steps.

543a287

Merge branch 'master' into bpo-30386

88c9e5f

Update the native interpreter build on Setup.dist and version changes.

408c05d

Add build a distribution for an application on a device

7a2fc54

xdegaye requested a review from tiran as a code owner August 8, 2017 16:50

xdegaye removed the request for review from tiran August 8, 2017 21:50

xdegaye added 8 commits August 9, 2017 19:45

The device/emulator distributions include now only the shared librari…

3161893

…es from the external libraries

The WITH_LIBRARIES environment variable selects the external libraries

648f3a9

Add the whatsnew and Misc/NEWS.d entries

fbedffc

Merge remote-tracking branch 'origin/master' into bpo-30386

d78ff4e

Merge remote-tracking branch 'origin/master' into bpo-30386

3cf267a

Fix a regression from 3161893

23172d8

terminfo/l/linux is now again installed on the emulator.

Patch libffi to set FFI_MMAP_EXEC_WRIT in configure

fde2fc7

See libffi/libffi#265. The external libraries are now untared in the build tree instead of the source tree previously.

Java 8 is now required to create an avd.

71a45d6

This is at Android build-tools version 25.0.3 and emulator version 26.0.3.

xdegaye added 5 commits November 20, 2017 16:36

Build ctypes with the headers of the cross-compiled libffi

863a083

Disable _ctypes on x86_64 (all long double tests fail)

21fd663

Disable uuid, Android has no uuid/uuid.h header

fd259a3

Disable grp, some grp.h functions are __INTRODUCED_IN_FUTURE

61d0cf5

Disable _crypt, Android does not have crypt.h

ab0e4ef

Merge remote-tracking branch 'origin/master' into bpo-30386

47614b4

vstinner requested changes Dec 12, 2017

View reviewed changes

bedevere-bot added the awaiting changes label Dec 12, 2017

There is now only one method to build Python

dbbca8b

xdegaye added 7 commits January 31, 2018 20:41

Support openssl-1.1.0g on all architectures

96fa8c8

Support ctypes also on 64 bits architectures

1f0ce9d

Support pip

77a30e9

sys.prefix must not be located on /sdcard: pip adds a sys.prefix/bin/ directory and fails upon PermissionError raised when using the /sdcard file system.

Remove the Doc/whatsnew/3.7rst and Misc/NEWS.d entries

cb9f4e3

Set the python version in build.mk instead of the Makefile

f109722

Do not use a symlink for inputrc

5bbf5e2

Add the 'setup' target to build third-party extension modules

0a51515

This was referenced Sep 15, 2018

Try python3.7 with Google NDK kivy/python-for-android#1202

Closed

Proof of concept for Python 3 without CrystaX, plus API 27 support kivy/python-for-android#1375

Closed

xdegaye closed this Dec 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-30386: Add a build infrastructure for Android. #1629

bpo-30386: Add a build infrastructure for Android. #1629

xdegaye commented May 17, 2017 •

edited by bedevere-bot

Loading

AraHaan commented May 17, 2017 •

edited

Loading

xdegaye commented May 30, 2017

xdegaye commented Aug 8, 2017

pmp-p commented Nov 17, 2017

xdegaye commented Nov 17, 2017

vstinner Dec 12, 2017

vstinner Dec 12, 2017

xdegaye Dec 12, 2017

bedevere-bot commented Dec 12, 2017

vstinner commented Dec 12, 2017

vstinner commented Dec 12, 2017

vstinner commented Dec 12, 2017

vstinner commented Dec 12, 2017

pmp-p commented Dec 12, 2017

xdegaye commented Dec 12, 2017

xdegaye commented Dec 12, 2017

pmp-p commented Dec 12, 2017

xdegaye commented Dec 13, 2017

pmp-p commented Dec 13, 2017 •

edited

Loading

xdegaye commented Dec 14, 2017

pmp-p commented Dec 14, 2017 •

edited

Loading

pmp-p commented Jan 21, 2018 •

edited

Loading

inclement commented Sep 18, 2018 •

edited

Loading

xdegaye commented Sep 19, 2018

bpo-30386: Add a build infrastructure for Android. #1629

bpo-30386: Add a build infrastructure for Android. #1629

Conversation

xdegaye commented May 17, 2017 • edited by bedevere-bot Loading

AraHaan commented May 17, 2017 • edited Loading

xdegaye commented May 30, 2017

xdegaye commented Aug 8, 2017

pmp-p commented Nov 17, 2017

xdegaye commented Nov 17, 2017

vstinner Dec 12, 2017

Choose a reason for hiding this comment

vstinner Dec 12, 2017

Choose a reason for hiding this comment

xdegaye Dec 12, 2017

Choose a reason for hiding this comment

bedevere-bot commented Dec 12, 2017

vstinner commented Dec 12, 2017

vstinner commented Dec 12, 2017

vstinner commented Dec 12, 2017

vstinner commented Dec 12, 2017

pmp-p commented Dec 12, 2017

xdegaye commented Dec 12, 2017

xdegaye commented Dec 12, 2017

pmp-p commented Dec 12, 2017

xdegaye commented Dec 13, 2017

pmp-p commented Dec 13, 2017 • edited Loading

xdegaye commented Dec 14, 2017

pmp-p commented Dec 14, 2017 • edited Loading

pmp-p commented Jan 21, 2018 • edited Loading

inclement commented Sep 18, 2018 • edited Loading

xdegaye commented Sep 19, 2018

xdegaye commented May 17, 2017 •

edited by bedevere-bot

Loading

AraHaan commented May 17, 2017 •

edited

Loading

pmp-p commented Dec 13, 2017 •

edited

Loading

pmp-p commented Dec 14, 2017 •

edited

Loading

pmp-p commented Jan 21, 2018 •

edited

Loading

inclement commented Sep 18, 2018 •

edited

Loading