Python 2.7.11 & Updates (Only for testing purposes)

[opacam]

Python 2.7.11 & Updates

[This is only for testing purposes, because, as pointed by @kived, it modifies a lot of stuff, so very soon it will be created some small PR's that will be easier to review and test, so... THIS PR IS NOT TO MERGE and it will be closed soon]

The new python2 recipe supports optional dependencies: sqlite3, openssl and libffi

• Updates the sqlite3's version for pygame's bootstrap
• Updates the jpeg's version for pygame's bootstrap
• Updates the freetype's version for pygame's bootstrap
• Updates the pil's version, adding freetype's support and solves jpeg's issues
• Solves conflicts with the distributed android libraries (openssl, freetype, harfbuzz), the trick is to link against the static libraries instead of linking with the shared.
• Fixes the pygame build (solves issue Pygame missing include & link path #687)
• Solves PIL does not compile with freetype2 support (issue PIL does not compile with freetype2 support #413)
• The sdl recipe has been modified to avoid error not finding python (patch applied at build time)

Important notes:

• The last commits, changes the build for pygame bootstrap. The integrated libraries inside pygame bootstrap components, jpeg, freetype and sqlite3, will not be used anymore... instead we will use the recipes from python-for-android because they are more updated, allow us to have more control, simplifies the code used in some recipes and also solves some issues related with the mentioned libraries.

My Build Specs:

• Tested with my app (with a lot of dependencies), using pygame and sdl2 bootstraps
• Android's NDK version: r10e-rc4
• Api version: 14
• Build System: Debian Stretch (amd64)

Thank to:

• Many Thanks to users: @brussee and @kived for his comments to helped me to polish things and improve the code.
• Also thanks to @dessant to encourage me to try to update from my previous pull request...actually it was quite easy starting from my previous python recipe (2.7.9).
• Also thanks all the developers who make possible this great project

-Update:
After the latest commits and some testing, I can say that the PR for python-2.7.11 works as expected for pygame and sdl2 and it should compile and run for everyone (I hope so...)

Note: Would like to mention that the issues that i found with sdl2 were caused by my old kivy's config file and some button calls triggered by "on_press" event (this two caused that i receive duplicated events for almost all events in my app). After modifying the entry "input" for my kivy's config file and changing my calls "on_press" to "on_release"....all works as expected :))

[opacam]

SDL2'S UPDATE:
I managed to apply the same formula used in pygame bootstrap to update the sdl2 libraries to the last version and force the build system to use the recipes from python-for-android for jpeg, freetype and png. All is compiled fine and runs fine but ... I did not push this changes because I find that kivy is not running as expected:

• The kivy's scrollview is not working as expected, the touch is not passed from the item to the scrollview, meaning that you can only scroll if the touch is made into the scrollview without touching the item's area (Similar issues has been referenced in Touchscreen input with SDL2 bootstrap #516)
• Also I think that some touches/events are emitted twice, I'm not sure of that, but the thing is that, in my app, when I push a button, it shows some widget and when it is pushed again the widget should be undecover...what happens is that the widget is never shown (that make me thinks that some events are emitted twice). Also it happens to me that, my app indexes video files from disk, and this files are duplicated (this is not an SDL2 issue but maybe is related to kivy).
• Of course that the same app is working as expected with pygame bootstrap (with similar changes applied already pushed for pygame bootstrap). And the same issues explained before, occurs with the distributed external libraries alongside the sdl2 package (freetype, jpeg,and png). My thought is that this update will not affect to the sdl2 bootstrap in a bad way...but should I push this changes?

@inclement, @kived, @brussee and @dessant:

You showed some interest with this pull request...what do you think?

NOTE:: I recently notice that the user @brussee has made a pull request for updating the sdl2 libraries (pull #708, pull #709, pull #710 and pull #713), this is great!!! I go further and integrates jpeg, freetype, and png recipes into sdl2's bootstrap.

[Tungsteno74]

Sorry if this will be a stupid question but i'm new with github repository and i'm not very able to do thing with it.
The question is:
Is possible to use this pull request updates within buildozer?

I use it for build my apk and i like to test your updates.

thanks for your help.

[opacam]

I'm not using buildozer for my build system, but i found that you can use a custom python-for-android, just by editing your buildozer spec file and setting the variable "android.p4a_dir", so... You can install the python-for-android with pip (if you don't wont to make changes to the source code) or clone the repo and switch to the python-2.7.11 branch (if you want to make modifications):

The pip command:

• pip install https://github.com/opacam/python-for-android/archive/python-2.7.11.zip

To clone the repo and switch to python-2.7.11 branch:

• git clone https://github.com/opacam/python-for-android.git
• cd python-for-android
• git checkout python-2.7.11

Then you can edit your buildozer spec file and set the variable "android.p4a_dir" to point to the location of the recently installed or cloned python-for-android repository

I guess that this way you can test it within buildozer

[Tungsteno74]

Sorry if i give you even bother, but I've done:

pip install https://github.com/opacam/python-for-android/archive/python-2.7.11.zip

and then in the buildozer.spec i tried:

android.p4a_dir = /usr/local/lib/python2.7/dist-packages/pythonforandroid/
android.p4a_dir = /usr/local/lib/python2.7/dist-packages/pythonforandroid
android.p4a_dir = /usr/local/lib/python2.7/dist-packages/

this is the 2.7.11 package (i checked it) but in the logcat is always showed

04-20 01:02:53.804 24787-24802/? I/python: [INFO ] [Python ] v2.7.2 (default, Apr 5 2016, 01:27:52)

Thus, is there something wrong?

thanks

Error: Ignoring unknown package filter 'build-tools-24.0.0rc0'
Warning: The package filter removed all packages. There is nothing to install.
         Please consider trying to update again without a package filter.
# Check that aidl can be executed
# Search for Aidl
# Run '/home/valerio/.buildozer/android/platform/android-sdk-20/build-tools/23.0.1/aidl'
# Cwd None
# Android packages installation done.
# Check application requirements
# Run './distribute.sh -l'
# Cwd /usr/local/lib/python2.7/dist-packages/pythonforandroid
/bin/sh: 1: ./distribute.sh: not found
# Command failed: ./distribute.sh -l
# 
# Buildozer failed to execute the last command
# The error might be hidden in the log above this error
# Please read the full log, and search for it before
# raising an issue with buildozer itself.
# In case of a bug report, please add a full log with log_level = 2

[opacam]

I think that the problem is that, despite you has updated your python-for-android, buildozer is detecting that you already have a working distribution, the one you has builded with the old python-for-android...so... skips the build (as you noticed with the logcat you posted...the build date is april 5). You must delete/backup your old build files, that way buildozer will build all the stuff again with the updated python-for-android. I thing that you can remove the old files in two ways:

• with the command: "buildozer android clean"
  (Note buildozer command: I don't test this command but it shoulld work...but, honestly, I don't know how far will go this command...maybe it will remove all the android related stuff including the sdk and ndk)
• Another choice will be to remove/backup manually the python-for-android folder (the one with all the the compilations). My guess is that all the build system done for python for android is in the following path (as a hidden path inside your home directory): ".buildozer/android/platform/python-for-android/"

Note Buildozer spec: I think that the p4a's path for the buildozer spec file is one of this two: "android.p4a_dir = /usr/local/lib/python2.7/dist-packages/pythonforandroid/" or the same without the final slash

Good Luck!!!

[Tungsteno74]

ok, i've tried to copy distribute.sh in /usr/local/lib/python2.7/dist-packages/pythonforandroid/ folder and now instead of not found i get access denied.

The problem is that i can't find where is located the distribute.sh in the default distribution.

p.s.: but the distribute.sh isn't obsolete?

[opacam]

You're right, the distribute.sh file is not used anymore.

My suggestions:

• Check you buildozer spec file and make sure that the entry "android.branch = ..." is commented
• Try the command: "buildozer android update" (this will update the target dependencies)
• And then run your usual buildozer's build command

PD: Do you have the latest version of buildozer installed which enables support for the p4a's new toolchain?...if not ...first of all you must upgrade your buildozer

[Tungsteno74]

I have done all you suggest. I've removed the distribute.sh from python 2.7.11 and update it.

This is my last try:

valerio@valerio-virtual01-xubuntu:~/PROGETTI/opentest$ sudo -H buildozer android debug
# Check configuration tokens
Buildozer is running as root!
This is not recommended, and may lead to problems later.
Are you sure you want to continue [y/n]? y
# Ensure build layout
# Check configuration tokens
# Preparing build
# Check requirements for android
# Run 'dpkg --version'
# Cwd None
Debian "dpkg", programma di gestione dei pacchetti versione 1.17.5 (amd64).
Questo è software libero; consultare la GNU General Public Licence versione 2
o successiva per le condizioni di copia. NON c'è alcuna garanzia.
# Search for Git (git)
#  -> found at /usr/bin/git
# Search for Cython (cython)
#  -> found at /usr/local/bin/cython
# Search for Java compiler (javac)
#  -> found at /usr/lib/jvm/java-7-oracle/bin/javac
# Search for Java keytool (keytool)
#  -> found at /usr/lib/jvm/java-7-oracle/jre/bin/keytool
# Install platform
# Apache ANT found at /root/.buildozer/android/platform/apache-ant-1.9.4
# Android SDK found at /root/.buildozer/android/platform/android-sdk-20
# Android NDK found at /root/.buildozer/android/platform/android-ndk-r9c
# Check application requirements
# Run './distribute.sh -l'
# Cwd /usr/local/lib/python2.7/dist-packages/pythonforandroid/
/bin/sh: 1: ./distribute.sh: not found
# Command failed: ./distribute.sh -l
# 
# Buildozer failed to execute the last command
# The error might be hidden in the log above this error
# Please read the full log, and search for it before
# raising an issue with buildozer itself.
# In case of a bug report, please add a full log with log_level = 2

My buildozer version is:

sudo buildozer version
# Check configuration tokens
Buildozer is running as root!
This is not recommended, and may lead to problems later.
Are you sure you want to continue [y/n]? y
Buildozer 0.32dev

Is the right version?

You advise me to forget buildozer and try directly with P4A?

[opacam]

You have the latest published version of buildozer but probably without python-for-android toolchain support because your buildozer keeps trying to run the distrubute.sh command (check the commits of the buildozer project at https://github.com/kivy/buildozer/commits/master). If you want to be sure that you have support for the new toolchain just work with a cloned repo of the buildozer project or install it via pip command directly from the git repo (pip install https://github.com/kivy/buildozer/archive/master.zip, for you with "sudo").

About trying directly from p4a...juas, juas, juas.. I asked myself this question a long time ago...I tried buildozer but it din't work for me. It's an amazing tool that will solve a lot of troubles to configure the build system... but is in aplha state.. and it means that is not ready, yet, for a production system. Moreover, I already had my own build system configured and working. Be sure, that at some time, I will use buildozer, but now I don't have enough time to do it (right now I'm struggling to make a binary for my app for macos...with my hackintosh...jaja... packaging a python app is a pain in the ass...I think that for an apple system is worst...). Your case is different... you have buildozer configured and you have a succesful build of your apk, so, I will give another shoot: Try to update your buildozer, remove all the old stuff related with you python-for-android and try to build your apk again. If it does not work maybe you can try to use python-for-android directly, it is not so difficult, and you can take your current buildozer configuration to adapt to your system and avoid downloading ndk's android and sdk's again...that way you will have the choice to build from buildozer or directly from python-for-android

Advice: If I were you, I would take seriuosly the buildozer's warning about running this tools with root privileges, unless you use xubuntu only for build your apk, because you can easely mess up your os, if you don't know what are you doing, because any system upgrade can broke your build system. Have you considered to install a python's virtual to develope your apk? That way you will keep your python environment isolated...of course...it doesn't matter if you use another os as your regular os

PD: I think that this conversation is in the wrong place...the issues you have right now are related with buildozer project and I'm not the right person to help you with this, not because I don't wanna help you but my knowledge of buildozer is very poor, and you can be sure that the programmers contributing with the buildozer project, can solve this problem quicker and better than me.
Anyway, hope you will solve the buildozer problem with my last indications

The force be with you!!!

[Tungsteno74]

You're right, I'm sorry if I made you waste time. You've been very helpful.
Many thanks for the help you gave me.

[opacam]

You don't have to apologize to me, glad to help you on whatever I can...this is only to keep the things ordered, despite I'm newbie at contributing to the p4a project, I'm pretty sure that the mantainers of this project, would like to keep all the things in his place because the kivy project is a big Project with a lot of subprojects as p4a or buildozer.

So...Please...don't doubt to write if you find something wrong, or maybe something that I wrote need some explanation or anything related with this pull request...

I hope You will solved the buildozer's quickly issues, maybe you are in the right way already...i hope so.

If you still have troubles to fix the issues with buildozer, I would like to suggest to create a new issue whitin the buildozer's project, so this way we will keep the things ordered and, probably, you will be on the road faster

So many thanks to be interested in this pull request, and I would like to to see you soon in this conversation to tell me that you have solved your problem and this new python recipe works well for you.

¡¡¡My best wishes and cheers!!!

[kived]

Thank you for all your work on this, however, I don't see this PR getting merged in its current state. It does way too much, much more than just updating the Python version. It really needs to be split into separate PRs so they can be reviewed and tested individually. Best practice is to keep each PR as small and specific as possible.

If some parts of this work require the updated Python version, just note as such in the new PR and link to the Python PR.

[opacam]

Ok, I will keep that in mind for future PR's. About this Pr... I will try to split it into several small PR to make it easy to review and test the changed code (I have already started the process of splitting the PR)

For now, I will keep this open for those who want to test the whole package. I will close this PR when created the new ones.