Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3 recipe #1412

Merged
merged 50 commits into from
Nov 10, 2018
Merged
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
dc8619b
Added python3 and hostpython3 files
inclement Sep 22, 2018
008b41d
Added hostpython3 recipe
inclement Sep 30, 2018
901a153
Got python3 recipe to the point of configure working
inclement Oct 14, 2018
8f8a050
Progressed the python3 recipe enough that python3 can be built
inclement Oct 14, 2018
a99478d
Added a temporary hardcoded APP_PLATFORM setting
inclement Oct 14, 2018
43ce635
Made other recipe types build properly when using python3
inclement Oct 14, 2018
3a09e10
Made bootstrap collation work with python3 recipe
inclement Oct 15, 2018
4a78c13
Fixed bootstrap code to correctly load and run a python3 bundle
inclement Oct 16, 2018
1d22bdd
Did initial implementation of an ndk-api build target option
inclement Oct 17, 2018
9491855
Made bootstraps check the ndk api against the requested minsdk
inclement Oct 18, 2018
9423cc8
Fixed APP_PLATFORM setting
inclement Oct 18, 2018
545aa02
Moved from bpo-30386 branch to python 3.7.0 official distribution
inclement Oct 18, 2018
5a4d443
Made all version references in python url dynamic
inclement Oct 18, 2018
24a57be
Began moving python bundle creation to the Python recipe
inclement Oct 19, 2018
0e0036a
Moved python3crystax python bundling to the recipe class
inclement Oct 19, 2018
bd38265
Moved python2 python bundle creation to python recipe
inclement Oct 19, 2018
a0838b6
Applied tito's fixes for python recipes installed via pip
inclement Oct 20, 2018
0f21372
Fixed webbrowser registration for python3
inclement Oct 20, 2018
4a094d8
Updated python3 recipe target version to 3.7.1
inclement Oct 20, 2018
d0f0113
Made __ANDROID_API__ take the same value as APP_PLATFORM
inclement Oct 20, 2018
b58febc
Corrected NDK platform finding to use ndk_api instead of android_api
inclement Oct 23, 2018
d419f62
Added include_root and link_root to python3 and python3crystax
inclement Oct 23, 2018
1ebdcee
Introduced some preliminary stdlib pruning during zip packaging
inclement Oct 23, 2018
d786718
Refactored out function for getting valid filens from a folder
inclement Oct 24, 2018
4964ec2
Made site-packages copying exclude some files/folders
inclement Oct 26, 2018
ec7ecac
Deleted some old commented out code
inclement Oct 26, 2018
a80fc6f
Made create_python_bundle return the site-packages dir
inclement Oct 26, 2018
d2435d6
Restructured dist_info.json creation
inclement Oct 26, 2018
07ae059
Removed hardcoded python version from sdl2 recipe
inclement Oct 26, 2018
3932fe2
Removed hardcoded api 21 in python3 recipe
inclement Oct 26, 2018
b25074e
Fixed python2 build following recent changes for python3
inclement Oct 27, 2018
82c1fc9
Added more (and clearer) checks for ndk-api vs android-api
inclement Oct 27, 2018
dbf9815
Fixed dist selection to account for ndk_api
inclement Oct 27, 2018
a963112
Improved code style per review comments
inclement Oct 27, 2018
b3cd815
Fixed typo in variable name
inclement Oct 27, 2018
f113131
Cleaned up code per review comments
inclement Nov 6, 2018
9cd161e
Made flake8 fixes
inclement Nov 7, 2018
462b1e5
Fixed reportlab recipe tests
inclement Nov 7, 2018
f8cc0e9
Added E129 to ignored list
inclement Nov 7, 2018
f7038f8
Made ndk-api default more intelligent
inclement Nov 8, 2018
da98194
Added default ndk-api settings to the testapps
inclement Nov 8, 2018
e48d82e
Updated travis.yml to target python3crystax testapp
inclement Nov 8, 2018
47a8daf
Updated documentation to talk about Python 3 support
inclement Nov 8, 2018
54c5b38
Improved handling of --android-api and --ndk-api
inclement Nov 8, 2018
c817ad0
Restructured discovery of android-api and ndk-api inputs
inclement Nov 8, 2018
f74ce04
Updated CI scripts to run a python3 testapp
inclement Nov 8, 2018
12346d2
Added ndk-api for sqlite,openssl testapp
inclement Nov 8, 2018
f150014
Added path to hostpython in python3 build env
inclement Nov 10, 2018
82ad9e5
Updated python3 testapp ndk-api to 21
inclement Nov 10, 2018
b076cd9
Removed Python3 testapp from CI for now
inclement Nov 10, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Began moving python bundle creation to the Python recipe
inclement committed Nov 6, 2018

Partially verified

This commit is signed with the committer’s verified signature.
spydon’s contribution has been verified via GPG key.
We cannot verify signatures from co-authors, and some of the co-authors attributed to this commit require their commits to be signed.
commit 24a57be57a986294c42e52973230c8a845a95657
49 changes: 2 additions & 47 deletions pythonforandroid/bootstraps/sdl2/__init__.py
Original file line number Diff line number Diff line change
@@ -107,53 +107,8 @@ def run_distribute(self):
shprint(sh.rm, '-rf', 'config/python.o')

elif self.ctx.python_recipe.name == 'python3':
ndk_dir = self.ctx.ndk_dir
py_recipe = self.ctx.python_recipe

## Build the python bundle:

# Bundle compiled python modules to a folder
modules_dir = join(python_bundle_dir, 'modules')
ensure_dir(modules_dir)

modules_build_dir = join(
self.ctx.python_recipe.get_build_dir(arch.arch),
'android-build',
'build',
'lib.linux-arm-3.7')
module_filens = (glob.glob(join(modules_build_dir, '*.so')) +
glob.glob(join(modules_build_dir, '*.py')))
for filen in module_filens:
shprint(sh.cp, filen, modules_dir)

# zip up the standard library
stdlib_zip = join(self.dist_dir, python_bundle_dir, 'stdlib.zip')
with current_directory(
join(self.ctx.python_recipe.get_build_dir(arch.arch),
'Lib')):
shprint(sh.zip, '-r', stdlib_zip, *os.listdir())

# copy the site-packages into place
shprint(sh.cp, '-r', self.ctx.get_python_install_dir(),
join(python_bundle_dir, 'site-packages'))

# copy the python .so files into place
python_build_dir = join(py_recipe.get_build_dir(arch.arch),
'android-build')
shprint(sh.cp, join(python_build_dir, 'libpython3.7m.so'),
'libs/{}'.format(arch.arch))
shprint(sh.cp, join(python_build_dir, 'libpython3.7m.so.1.0'),
'libs/{}'.format(arch.arch))

info('Renaming .so files to reflect cross-compile')
site_packages_dir = join(python_bundle_dir, 'site-packages')
py_so_files = shprint(sh.find, site_packages_dir, '-iname', '*.so')
filens = py_so_files.stdout.decode('utf-8').split('\n')[:-1]
for filen in filens:
parts = filen.split('.')
if len(parts) <= 2:
continue
shprint(sh.mv, filen, parts[0] + '.so')
self.ctx.python_recipe.create_python_bundle(
join(self.dist_dir, python_bundle_dir), arch)

elif self.ctx.python_recipe.from_crystax: # Python *is* loaded from crystax
ndk_dir = self.ctx.ndk_dir
18 changes: 11 additions & 7 deletions pythonforandroid/recipe.py
Original file line number Diff line number Diff line change
@@ -1112,14 +1112,18 @@ def include_root(self, arch):
def link_root(self):
raise NotImplementedError('Not implemented in TargetPythonRecipe')

# @property
# def ctx(self):
# return self._ctx
@property
def major_minor_version_string(self):
from distutils.version import LooseVersion
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we lazy import for a specific reason, we may want to add a comment telling why. If we don't need to lazy import then maybe we want to move this up?

return '.'.join([str(v) for v in LooseVersion(self.version).version[:2]])

# @ctx.setter
# def ctx(self, ctx):
# self._ctx = ctx
# ctx.python_recipe = self
def create_python_bundle(self, dirn, arch):
"""
Create a packaged python bundle in the target directory, by
copying all the modules and standard library to the right
place.
"""
raise NotImplementedError('{} does not implement create_python_bundle'.format(self))


def md5sum(filen):
56 changes: 54 additions & 2 deletions pythonforandroid/recipes/python3/__init__.py
Original file line number Diff line number Diff line change
@@ -4,8 +4,9 @@
check_all, is_api_lt, is_ndk)
from pythonforandroid.logger import logger
from pythonforandroid.util import ensure_dir
from os.path import exists, join, realpath
from os import environ
from os.path import exists, join, realpath, split
from os import environ, listdir
import glob
import sh


@@ -111,5 +112,56 @@ def include_root(self, arch_name):
def link_root(self, arch_name):
return join(self.get_build_dir(arch_name),
'android-build')

def create_python_bundle(self, dirn, arch):
ndk_dir = self.ctx.ndk_dir

# Bundle compiled python modules to a folder
modules_dir = join(dirn, 'modules')
ensure_dir(modules_dir)

modules_build_dir = join(
self.get_build_dir(arch.arch),
'android-build',
'build',
'lib.linux-arm-3.7')
module_filens = (glob.glob(join(modules_build_dir, '*.so')) +
glob.glob(join(modules_build_dir, '*.py')))
for filen in module_filens:
shprint(sh.cp, filen, modules_dir)

# zip up the standard library
stdlib_zip = join(dirn, 'stdlib.zip')
with current_directory(
join(self.get_build_dir(arch.arch),
'Lib')):
shprint(sh.zip, '-r', stdlib_zip, *listdir('.'))

# copy the site-packages into place
shprint(sh.cp, '-r', self.ctx.get_python_install_dir(),
join(dirn, 'site-packages'))

# copy the python .so files into place
python_build_dir = join(self.get_build_dir(arch.arch),
'android-build')
shprint(sh.cp,
join(python_build_dir,
'libpython{}m.so'.format(self.major_minor_version_string)),
'libs/{}'.format(arch.arch))
shprint(sh.cp,
join(python_build_dir,
'libpython{}m.so.1.0'.format(self.major_minor_version_string)),
'libs/{}'.format(arch.arch))

info('Renaming .so files to reflect cross-compile')
site_packages_dir = join(dirn, 'site-packages')
py_so_files = shprint(sh.find, site_packages_dir, '-iname', '*.so')
filens = py_so_files.stdout.decode('utf-8').split('\n')[:-1]
for filen in filens:
file_dirname, file_basename = split(filen)
parts = file_basename.split('.')
if len(parts) <= 2:
continue
shprint(sh.mv, filen, join(file_dirname, parts[0] + '.so'))

recipe = Python3Recipe()