load ssl and crypto libs before loading python, for (4.1.2 or ear…

[akshayaurora]

…lier devices)

closes #866

[dkruk]

I tried it on Android Virtual Device and unfortunately it doesn't work for me.
I used Android 4.1(API=16) and Android 4.2(API=17) for checking.

[inclement]

@akshayaurora I feel that I lost track of this issue - is this still a good fix for a current bug?

[rnixx]

should cffi be considered here as well? https://github.com/kivy/python-for-android/pull/903/files#diff-58dede73f12c2d1e35c61afbe5f187aaR25

[rnixx]

and what about other potential libs? does it make sence to have auto detection of everything contained in lib/armeabi-v7a/ in the apk?

[akshayaurora]

@inclement yes, this is a working fix for the current issue on older androids.

@rnixx I would not be comfortable with loading all available libs in that folder... that would mean malicious could be easily injected... is there a easier workaround for this?

cffi could definitely be included here https://github.com/kivy/python-for-android/pull/903/files#diff-58dede73f12c2d1e35c61afbe5f187aaR25

[akshayaurora]

@dzmitrykrukouski the app would crash on emulator cause of missing OpenGL... please test on actual device.

[rnixx]

@akshayaurora It would be great if libs to load get extracted from installed recipes if there are extensions. Whether there is something to load can be defined on the related recipes. It might be just a text file generated somewhere and read in https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android/PythonUtil.java.

What do you think?

Btw - libs are loaded in https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/webview/build/src/org/kivy/android/PythonUtil.java as well. Maybe the final mechanism needs to be used here too.

[rnixx]

There's one more https://github.com/kivy/python-for-android/blob/master/pythonforandroid/bootstraps/service_only/build/src/org/kivy/android/PythonUtil.java

[dkruk]

@akshayaurora unfortunately I don't have actual device with Android 4.2 or earlier.

[rnixx]

Some updates

• I tested with a Samsung Galaxy Nexus with Android 4.2.1
• After building my App with the changes from this PR + adding ffi to lib list I ran into a linker error when trying to load _cffi_backend.so from cryptography -> symbol not found "getline". So apart from loading libs from default path there are also problems with compiled extensions, at least the pointed one.
• Buildozer currently installs NDK r9c by default. I read somewhere in the wild that updating the NDK might fix the linking problems. I tried with r11c and r13b, but apart that buildozer cannot download newer NDK's due to a changed download URL, the error re-appeared. Just got back the original "Couldn't load python3.5m: findLibrary returned null" error (where I started searching from). Even worse, if commenting out "python3.5m" in PythonUtil.java loading python2.7 failed as well - again with some symbols not found.
• Finally I tested with NDK r9c on an Android 6.0 device. The App starts there - but with some glitches accessing native stuff via pyjnius. But that has nothing to do with this issue.

[akshayaurora]

@rnixx could you please put up a example that uses needs, uses cffi? I could test with including it.

[akshayaurora]

loading python2.7 failed as well - again with some symbols not found.

@rnixx could you paste that log, would be interesting to see which symbols were not found.

[rnixx]

@akshayaurora ad example -> just add python2, cryptography and pyopenssl to your recipes.

When starting App which has been built with NDK r13b and commented out python3.5m here master...rnixx:master#diff-58dede73f12c2d1e35c61afbe5f187aaR42, adb logcat tells:

V/PythonActivity( 6245): About to do super onCreate
V/SDL ( 6245): Device: maguro
V/SDL ( 6245): Model: Galaxy Nexus
V/SDL ( 6245): onCreate():null
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2.so 0x4185c720
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2_image.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2_image.so 0x4185c720
D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libSDL2_image.so 0x4185c720, skipping init
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2_mixer.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2_mixer.so 0x4185c720
D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libSDL2_mixer.so 0x4185c720, skipping init
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libSDL2_ttf.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libSDL2_ttf.so 0x4185c720
D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libSDL2_ttf.so 0x4185c720, skipping init
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libssl1.0.2h.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libssl1.0.2h.so 0x4185c720
D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libssl1.0.2h.so 0x4185c720, skipping init
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libcrypto1.0.2h.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libcrypto1.0.2h.so 0x4185c720
D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libcrypto1.0.2h.so 0x4185c720, skipping init
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libffi.so 0x4185c720
D/dalvikvm( 6245): Added shared lib /data/app-lib/org.foo.myapp/libffi.so 0x4185c720
D/dalvikvm( 6245): No JNI_OnLoad found in /data/app-lib/org.foo.myapp/libffi.so 0x4185c720, skipping init
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libpython2.7.so 0x4185c720
E/dalvikvm( 6245): dlopen("/data/app-lib/org.foo.myapp/libpython2.7.so") failed: Cannot load library: soinfo_relocate(linker.cpp:975): cannot locate symbol "wait4" referenced by "libpython2.7.so"...
D/dalvikvm( 6245): Trying to load lib /data/app-lib/org.foo.myapp/libmain.so 0x4185c720
E/dalvikvm( 6245): dlopen("/data/app-lib/org.foo.myapp/libmain.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libpython2.7.so" needed by "libmain.so"; caused by find_library(linker.cpp:889): "libpython2.7.so" failed to load previously
W/System.err( 6245): Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libpython2.7.so" needed by "libmain.so"; caused by find_library(linker.cpp:889): "libpython2.7.so" failed to load previously

As said above, when leaving python3.5m in the list I get the error message that this one is not found

[rnixx]

This is the error when building with NDK r9c NOT trying to load _cffi_backend.so manually like so master...rnixx:master#diff-58dede73f12c2d1e35c61afbe5f187aaR80:

I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/app/myapp/messaging.py", line 22, in
I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/app/twisted/internet/ssl.py", line 59, in
I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/OpenSSL/init.py", line 8, in
I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in
I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/OpenSSL/_util.py", line 3, in
I/python ( 6652): File "/home/me/workspace/myapp/.buildozer/android/platform/build/dists/myapp/private/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in
I/python ( 6652): ImportError: Cannot load library: find_library(linker.cpp:889): "/data/data/at.r2solutions.r2_client/files/app/lib/python2.7/site-packages/_cffi_backend.so" failed to load previously

You need to import from OpenSSL import SSL to provoke this.

[rnixx]

And this happens if I try to load _cffi_backend.so manually in PythonUtil.java at startup:

D/dalvikvm( 7370): Trying to load lib /data/data/org.foo.myapp/files/app/lib/python2.7/site-packages/_cffi_backend.so 0x4185c8f8
E/dalvikvm( 7370): dlopen("/data/data/org.foo.myapp/files/app/lib/python2.7/site-packages/_cffi_backend.so") failed: Cannot load library: soinfo_relocate(linker.cpp:975): cannot locate symbol "getline" referenced by "_cffi_backend.so"...

[rnixx]

@akshayaurora Do you need more / other details?

[akshayaurora]

@rnixx nope, that's enough details...I think I have seen the getline issue in a older version...searching...

[akshayaurora]

just for reference http://stackoverflow.com/questions/13112784/undefined-reference-to-getline-in-c

[frankgould]

I was able to resolve the following error (pasted below) by updating 5 copies of PythonUtil.java after searching for them in .buildozer. I inserted the two lines from the github link below the error, in these 5 files and my Android 4.2.2 tablet ran the app flawlessly. I have had this problem since September last year and am very happy KeyWeeUsr was able to guide to the solution, thank you very much.

E/dalvikvm( 3869): dlopen("/data/app-lib/com.masterpics.slideshow-1/libpython2.7.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libssl1.0.2h.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libssl1.0.2h.so" not found

c1659b8

I probably should go back to see if I can remove this code above to see if it is unnecessary in some of the PythonUtil.java files. Below are the list I updated:
.buildozer/android/platform/build/build/bootstrap_builds/sdl2-python2/src/org/kivy/android
.buildozer/android/platform/build/dists/slideshow/src/org/kivy/android
.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android
.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/service_only/build/src/org/kivy/android
.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/webview/build/src/org/kivy/android

Thank you all!

[Zen-CODE]

Using this path, I get:

SDL Error.
Error: Couldn't load python3.5m from load dalvik.system.PathClassLoader[DexPathList[[zip_file "/data/app/xxx.apk"],nativeLibraryDirectories=[/data/app-lib/xxx, /vendor/lib, /system/lib]]]:
findLibrary returned Null

If I comment out the python3.5m from line 18 of the PR, I get

SDL Error
And error occurred while trying to start the application. Please try again and/or re-install
Error: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libpython2.7.so" needed by "main.so"; caused by find_library(linker.cpp:889): "libpython2.7.so" filed to load previously.

We are using 2.7, and the APK works on Andorid > 4.4.

[inclement]

@Zen-CODE I've just come around to investigating this, and it's an annoyance that our library loader actually obscures the real errors. If you're still looking at it, could you change the PythonActivity code to:

	public static void loadLibraries(File filesDir) {

        String filesDirPath = filesDir.getAbsolutePath();
        boolean foundPython = false;

		for (String lib : getLibraries()) {
            Log.v("python", "Loading library: " + lib);
		    try {
                System.loadLibrary(lib);
                if (lib.startsWith("python")) {
                    foundPython = true;
                }
            } catch(UnsatisfiedLinkError e) {
                // If this is the last possible libpython
                // load, and it has failed, give a more
                // general error
                Log.v("python", "Library loading error: " + e.getMessage());
                if (lib.startsWith("python3.6") && !foundPython) {
                    throw new java.lang.RuntimeException("Could not load any libpythonXXX.so");
                } else if (lib.startsWith("python")) {
                    continue;
                } else {
                    Log.v("python", "An UnsatisfiedLinkError occurred loading " + lib);
                    throw e;
                }
            }
        }

        try {
            System.load(filesDirPath + "/lib/python2.7/lib-dynload/_io.so");
            System.load(filesDirPath + "/lib/python2.7/lib-dynload/unicodedata.so");
        } catch(UnsatisfiedLinkError e) {
            Log.v(TAG, "Failed to load _io.so or unicodedata.so...but that's okay.");
        }
        
        try {
            // System.loadLibrary("ctypes");
            System.load(filesDirPath + "/lib/python2.7/lib-dynload/_ctypes.so");
        } catch(UnsatisfiedLinkError e) {
            Log.v(TAG, "Unsatisfied linker when loading ctypes");
        }

        Log.v(TAG, "Loaded everything!");
    }

This should print some more helpful information about what has gone wrong. I think I have it working now, but am not yet sure because it still fails for an emulator-related reason. I did have to work through a couple of unexpected problems, including that building for api 19 didn't work (while 15 appears to).

If I'm right about having it working now, I'll fix up this PR and propose a new one for people to test within the next few days. The method here seems solid though. I'm sorry it's taken so long to get around to it, I haven't tackled it before due to the lack of devices with this issue - I would have merged it if it clearly worked for everyone, but as above it has seemed not quite there.

[Zen-CODE]

Insert the above code in PythonActivity.java still gives me this in the logcat.

V/PythonActivity( 8502): Data version is 1503657024.2
I/SurfaceFlinger(  476): EventThread Client Pid (8502) created
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2.so 0x4133b4a0
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2_image.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2_image.so 0x4133b4a0
D/dalvikvm( 8502): No JNI_OnLoad found in /data/app-lib/xxx-1/libSDL2_image.so 0x4133b4a0, skipping init
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2_mixer.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2_mixer.so 0x4133b4a0
D/dalvikvm( 8502): No JNI_OnLoad found in /data/app-lib/xxx-1/libSDL2_mixer.so 0x4133b4a0, skipping init
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libSDL2_ttf.so 0x4133b4a0
D/dalvikvm( 8502): Added shared lib /data/app-lib/xxx-1/libSDL2_ttf.so 0x4133b4a0
D/dalvikvm( 8502): No JNI_OnLoad found in /data/app-lib/xxx-1/libSDL2_ttf.so 0x4133b4a0, skipping init
D/dalvikvm( 8502): Trying to load lib /data/app-lib/xxx-1/libpython2.7.so 0x4133b4a0
E/dalvikvm( 8502): dlopen("/data/app-lib/xxx-1/libpython2.7.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found
W/System.err( 8502): Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found
D/skia    ( 8502): Flag is not 10

That being said, I see there are multiple PythonActivity.java files. Which one should be changing?

[Zen-CODE]

Also, building with "android.api = 15" still produces the same error.

[inclement]

@Zen-CODE My best guess is that your error represents the same problem as openssl, but with libsqlite3.so instead, in which case it could be fixed by adding "sqlite3" to the module loading list. I intend to test sqlite3 myself shortly to confirm this.

I'll push my changes to the logging to the master branch shortly, to make it easier to use.

Sorry about being unclear about where to apply the changes, the best thing to do is to clean any existing distributions and edit the version in pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android. I'll try to add some proper documentation about this.

[inclement]

I've just pushed my logging improvements to master, so you should be able to get them by pulling that. I'm now investigating sqlite3.

[inclement]

@Zen-CODE I've made PR #1106 with my rebase plus restructuring. Does that work for you?

[Zen-CODE]

sqlite3 is in the requirements. Get this error on screen:

SDL Error 
Error: cannot load library:
soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library (linker.cpp: 745): library "libsqlite3.so" not found.

And the log has this:

I/SurfaceFlinger(  130): EventThread Client Pid (6949) created
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2.so 0x4176f440
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_image.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_image.so 0x4176f440
D/dalvikvm( 6949): No JNI_OnLoad found in /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_image.so 0x4176f440, skipping init
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_mixer.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_mixer.so 0x4176f440
D/dalvikvm( 6949): No JNI_OnLoad found in /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_mixer.so 0x4176f440, skipping init
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_ttf.so 0x4176f440
D/dalvikvm( 6949): Added shared lib /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_ttf.so 0x4176f440
D/dalvikvm( 6949): No JNI_OnLoad found in /data/app-lib/speedtest.maths.camiweb.com-1/libSDL2_ttf.so 0x4176f440, skipping init
D/dalvikvm( 6949): Trying to load lib /data/app-lib/speedtest.maths.camiweb.com-1/libpython2.7.so 0x4176f440
E/dalvikvm( 6949): dlopen("/data/app-lib/speedtest.maths.camiweb.com-1/libpython2.7.so") failed: Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found
W/System.err( 6949): Cannot load library: soinfo_link_image(linker.cpp:1635): could not load library "libsqlite3.so" needed by "libpython2.7.so"; caused by load_library(linker.cpp:745): library "libsqlite3.so" not found

Complete log here: https://gist.github.com/Zen-CODE/79998bb0a569374dde008ab58b8bc3da

[inclement]

@Zen-CODE Just to be clear, by 'module loading list' I mean the list of strings in PythonUtil.java, not the buildozer.spec requirements.

I believe my PR at #1106 should fix this issue. At the very least, it should give an extended log about what it finds (or fails to find) when trying to load sqlite3.

[Zen-CODE]

The log above was generated using PR #1106 changes.

[inclement]

@Zen-CODE Are you sure it's using the new code? There really should now be more logs than that. Could you paste the full logcat output?

[Zen-CODE]

The log above was generated using PR #1106 changes, but let me do another one.

So, I am changing the file:
/.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android/PythonUtil.java

To verify this is the one being used, I do a clean and build and the (deliberate) syntax error I inserted crashes it, so it is using the file. When I use the new version from PR 1106, it crashes with the same libsqlite3 issue, full output here: https://gist.github.com/Zen-CODE/cdca11cb6e57aab7e2962ffd876597b6

[inclement]

@Zen-CODE I still can't work out why you wouldn't be seeing the more extended logs - in particular, the line at https://github.com/kivy/python-for-android/pull/1106/files#diff-58dede73f12c2d1e35c61afbe5f187aaR58 should be called for every library, printing output using the "pythonutil" tag. Do you see anything I'm missing here (maybe I'm mistaken!).

The actual dist you've built should be in the .buildozer somewhere - I think .buildozer/android/platform/build. Could you check that the PythonUtil in your dist matches the changes in the main file at /.buildozer/android/platform/python-for-android-master/pythonforandroid/bootstraps/sdl2/build/src/org/kivy/android/PythonUtil.java? Assuming so, would you be able to zip up and send me the dist you find there? I think I should be able to import it and try to investigate from there.

[Zen-CODE]

@inclement. Sorry, it just occurred to me that perhaps it the way I capture the log using the "> log.txt" post-fix.

buildozer android debug depoly run logcat > log.txt

Perhaps the logging you are expecting to not redirect? Or how should I generate the log?

[inclement]

I'm closing this as superseded by #1106 (now merged), but I'm still not clear if it has fixed the issue for everyone. Feel free to open a new issue if you still experience this problem with the master branch.