Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding AssetFinder subdirectories to sys.path causes extract_packages error #820

Closed
mhsmith opened this issue Mar 3, 2023 · 5 comments · Fixed by #924
Closed

Adding AssetFinder subdirectories to sys.path causes extract_packages error #820

mhsmith opened this issue Mar 3, 2023 · 5 comments · Fixed by #924
Labels
bug
Milestone
15.0

Comments

@mhsmith
Copy link
Member

mhsmith commented Mar 3, 2023

From #818:

>>> from oct2py import octave
./java/android/importer.py:554: DeprecationWarning: Jupyter is migrating its paths to use standard platformdirs
given by the platformdirs library.  To remove this warning and
see the appropriate new directories, set the environment variable
`JUPYTER_PLATFORM_DIRS=1` and then run `jupyter --paths`.
The use of platformdirs will be the default in `jupyter_core` v6
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/oct2py/__init__.py", line 27, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/oct2py/core.py", line 14, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/metakernel/__init__.py", line 2, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/metakernel/_metakernel.py", line 22, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/ipykernel/kernelapp.py", line 52, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/ipykernel/ipkernel.py", line 23, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/ipykernel/debugger.py", line 40, in <module>
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/ipykernel/debugger.py", line 23, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/debugpy/server/__init__.py", line 7, in <module>
  File "import.pxi", line 26, in java.chaquopy.import_override
  File "/data/data/com.chaquo.python.pkgtest3/files/chaquopy/AssetFinder/requirements/debugpy/_vendored/force_pydevd.py", line 32, in <module>
  File "stdlib/importlib/__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1407, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1379, in _get_spec
  File "./java/android/importer.py", line 425, in find_spec
  File "./java/android/importer.py", line 449, in find_module
  File "./java/android/importer.py", line 476, in extract_dir
AttributeError: 'AssetFinder' object has no attribute 'extract_packages'

The problem is in the following code:

                    if (infix == "/__init__") and ("." not in mod_name):
                        # This is a top-level package: extract all data files within it.
                        self.extract_dir(prefix)

Because of the "." not in mod_name check, extract_dir will only be called on a finder which is directly on sys.path. Normally this would mean its path is directly below AssetFinder, which are the only finders that currently have an extract_packages attribute. However, force_pydevd.py uses a context manager to temporarily add a subdirectory to sys.path, so it can import a vendored package without using a prefix.

We should be able to support this use case. We may be able to fix it simply by copying extract_packages from the parent finder, or somehow ensuring that extract_dir is called on the root finder.
@mhsmith mhsmith added the bug label Mar 3, 2023
@mhsmith mhsmith added this to the 14.1 milestone Mar 3, 2023
@mhsmith mhsmith mentioned this issue Mar 3, 2023
Closed
@mhsmith mhsmith mentioned this issue Mar 27, 2023
Closed
@mhsmith
Copy link
Member Author

mhsmith commented Mar 27, 2023

See also #754 (comment).

@ItsCubeTime ItsCubeTime mentioned this issue Apr 1, 2023
Closed
@mhsmith mhsmith removed this from the 15.0 milestone Jul 14, 2023
@dbnicholson
Copy link
Contributor

I'm attempting to use https://github.com/learningequality/kolibri, which vendors a whole bunch of stuff in its wheel and updates sys.path to use them at runtime. I hit the above error and can't find a way to workaround it. I see you removed this from the 15.0 milestone. Are there any plans to fix this?

@mhsmith
Copy link
Member Author

mhsmith commented Jul 20, 2023

Please post the full stack trace of your error, and I'll take a look at it.

@dbnicholson
Copy link
Contributor

Here's the stack trace:

07-20 15:31:35.753  7383  7383 E AndroidRuntime: FATAL EXCEPTION: main
07-20 15:31:35.753  7383  7383 E AndroidRuntime: Process: org.endlessos.testapp, PID: 7383
07-20 15:31:35.753  7383  7383 E AndroidRuntime: java.lang.RuntimeException: Unable to start activity ComponentInfo{org.endlessos.testapp/org.endlessos.testapp.MainActivity}: com.chaquo.python.PyException: AttributeError: 'AssetFinder' object has no attribute 'extract_packages'
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3635)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3792)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2210)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.os.Handler.dispatchMessage(Handler.java:106)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.os.Looper.loopOnce(Looper.java:201)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.os.Looper.loop(Looper.java:288)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.ActivityThread.main(ActivityThread.java:7839)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at java.lang.reflect.Method.invoke(Native Method)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1003)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: Caused by: com.chaquo.python.PyException: AttributeError: 'AssetFinder' object has no attribute 'extract_packages'
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.java.android.importer.extract_dir(importer.py:476)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.java.android.importer.find_module(importer.py:449)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.java.android.importer.find_spec(importer.py:425)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.importlib._bootstrap_external._get_spec(<frozen importlib._bootstrap_external>:1395)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.importlib._bootstrap_external.find_spec(<frozen importlib._bootstrap_external>:1423)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.importlib._bootstrap._find_spec(<frozen importlib._bootstrap>:925)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.importlib._bootstrap._find_and_load_unlocked(<frozen importlib._bootstrap>:982)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.importlib._bootstrap._find_and_load(<frozen importlib._bootstrap>:1007)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:20)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:26)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.kolibri.utils.server.<module>(server.py:13)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.java.chaquopy.import_override(import.pxi:26)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.main.get_url(main.py:28)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.chaquopy_java.call(chaquopy_java.pyx:354)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at <python>.chaquopy_java.Java_com_chaquo_python_PyObject_callAttrThrowsNative(chaquopy_java.pyx:326)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at com.chaquo.python.PyObject.callAttrThrowsNative(Native Method)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at com.chaquo.python.PyObject.callAttrThrows(PyObject.java:232)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at com.chaquo.python.PyObject.callAttr(PyObject.java:221)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at org.endlessos.testapp.MainActivity.onCreate(MainActivity.java:39)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.Activity.performCreate(Activity.java:8051)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.Activity.performCreate(Activity.java:8031)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1329)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3608)
07-20 15:31:35.753  7383  7383 E AndroidRuntime: 	... 12 more

What's happening in the python function is that it's trying to import kolibri.utils.server and that is trying to import requests. The issue is that kolibri bundles all of its dependencies in its dist directory and then prepends that directory sys.path. I'm using the latest prerelease wheel, but the stable version on pypi has the same setup.

This may sound insane (and I mostly agree), but it does work on android as the wheel bundles native libraries for all the android platforms. We're currently building our app with pythonforandroid and it works in that sense. The problem you're describing above where a directory in sys.path is not the root asset path seems to be exactly what is happening here. I.e., a bare requests is being imported and chaquopy is assuming that it must be in the root assets directory and therefore the root AssetFinder can be used.

@mhsmith
Copy link
Member Author

mhsmith commented Jul 21, 2023 •
edited
Loading

OK, I'll see if I can fix this in the next Chaquopy version.

Related: #917

@mhsmith mhsmith added this to the 15.0 milestone Jul 21, 2023
@mhsmith mhsmith mentioned this issue Jul 25, 2023
Merged
dbnicholson added a commit to dbnicholson/chaquopy that referenced this issue Aug 1, 2023
@dbnicholson
Fix bare imports in AssetFinder subdirectories
fe21bb7 
When creating a non-root AssetFinder, copy the parent's `extract_packages` so
that importing bare modules works correctly. This is typically needed when
packages are vendored and `sys.path` is dynamically updated to prefer the
vendored version.

Fixes: chaquo#820
@dbnicholson dbnicholson mentioned this issue Aug 1, 2023
Merged
@mhsmith mhsmith mentioned this issue Dec 18, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
15.0
Development

Successfully merging a pull request may close this issue.

Fix bare imports in AssetFinder subdirectories dbnicholson/chaquopy
2 participants
@mhsmith @dbnicholson