Error with dependency trying to open a file

[DiegoMarcia]

Chaquopy version

com.chaquo.python:gradle:9.0.0

Devices or emulators where the issue happens

Android emulator version 30.4.5.0 emulating Galaxy Nexus API 30

Hello,
I am working on a mobile app integrating the problog interpreter, which is written in python.
I included problog in my dependencies, and it generally works fine for basic operations.

However, I get an exception whenever I try to use one of problog's modules written in python: the problog engine doesn't find the required file.

here's the exception:

2021-05-19 18:24:42.070 11109-11131/my.app.test E/TestRunner: com.chaquo.python.PyException: ProbLogError: [Errno 2] No such file or directory: '/data/user/0/my.app.test/files/chaquopy/AssetFinder/requirements/problog/library/assert'.
        at <python>.problog.program.__init__(program.py:324)
        at <python>.problog.clausedb.consult(clausedb.py:716)
        at <python>.problog.clausedb.use_module(clausedb.py:763)
        at <python>.problog.engine_builtin._use_module(engine_builtin.py:1707)
        at <python>.problog.engine_stack.__call__(engine_stack.py:2500)
        at <python>.problog.engine_stack.__call__(engine_stack.py:2463)
        at <python>.problog.engine_stack.eval_default(engine_stack.py:1201)
        at <python>.problog.engine_stack.eval_builtin(engine_stack.py:1197)
        at <python>.problog.engine_stack.eval(engine_stack.py:212)
        at <python>.problog.engine_stack.eval_call(engine_stack.py:1077)
        at <python>.problog.engine_stack.eval(engine_stack.py:212)
        at <python>.problog.engine_stack.eval_clause(engine_stack.py:1134)
        at <python>.problog.engine_stack.eval(engine_stack.py:212)
        at <python>.problog.engine_stack.execute(engine_stack.py:590)
        at <python>.problog.engine._process_directives(engine.py:228)
        at <python>.problog.engine.prepare(engine.py:200)
        ...

No python file from problog's library can be found at that path, here's a comparison between what can be found in a virtualenv and on the emulator filesystem:

I have read the often-cited issue 144, but my problem here is slightly different: I'm not trying to read a file from my code, it is an installed dependency (problog) which is trying to load a python file from its own assets.

What can I do? Where did those python files get moved to?

Thank you.

[sarthaksharma058439]

Hey ! you can check out this video for read file in chaquopy, it is very helpful !!

[mhsmith]

@sarthaksharma058439: It's clear that the original poster is aware of the basics of data file access in Chaquopy, so this video is unhelpful. This is the second or third time you've done this, and I've warned you before. So from now on, please email me to ask for permission before posting a link to one of your videos. Otherwise I'll have to ban you from the tracker for wasting my users' time.

[mhsmith]

@DiegoMarcia: For performance, the current version of Chaquopy loads Python modules directly from the APK, and the only files it extracts to the device filesystem are those whose names have non-Python extensions. Actually, this is the first example I've seen of a library which expects to be able to read its own source code as a text file at runtime.

Here's a possible workaround:

• Prevent the .py files from being compiled to .pyc format by adding pyc { pip false } to your build.gradle file (see the documentation).

• Run the following code to extract assert.py using an internal importer function:

  import problog
  finder = problog.__loader__.finder
  finder.extract_if_changed("problog/library/assert.py")
  

  Make sure you use the top-level package, i.e. problog, not problog.library.

  You can do the same thing for any other .py files that you want to extract.

I can't test this fully because I don't know how to use problog, so if you need any more help, please provide some example code.

[DiegoMarcia]

Thank you very much @mhsmith for your help!

I found a way to use the module in its .pyc form, therefore keeping all python files still compiled.
(I'm not sure whether this has a real impact on performance, or if the overhead from my method cancels this gain)

Anyway, I think the whole method might be useful with another dependency (a Python wrapper for a shared library), so I was wondering:
is it possible to use the pyc { pip false } directive for specific dependencies only?

Thank you again!

[mhsmith]

Not at the moment: as I said, this is the first time I've seen anything that needed it.

[mhsmith]

Anyway, I think the whole method might be useful with another dependency (a Python wrapper for a shared library),

Chaquopy has built-in support for many of the ways that packages load shared libraries, so hopefully this method wouldn't be needed there. But the library would have to be compiled specifically for Android. If you need any help with that, please create a separate issue, but include a link to this one.

[eldon922]

Hi, can you elaborate more how to extract .py file? i have my own .py file and i need to have it in AssetFinder folder

[mhsmith]

Just follow the instructions in this comment, but replace the package name and filename with the one you want.