Meta issue for cleaning up import system cruft #97850

warsaw · 2022-10-04T22:12:32Z

This is a meta issue tracking all the things that need to be cleaned up in the import system. From removing long deprecated APIs to migrating internal access to __spec__, we'll have several task lists for each related set of clean ups.

@brettcannon @ericsnowcurrently for visibility.

Removing deprecated APIs

Over in What's New for Python 3.11 there's a long list of things that have been deprecated long enough that they can be removed in Python 3.12. Here's that task list for tracking purposes. Watch for linked PRs (maybe one big one or many small ones -- we'll see!).

@brettcannon @ericsnowcurrently for visibility.

Update docs

Make sure examples and details from PEP 302 are covered in the importlib docs (/cc @Yhg1s )
Remove https://docs.python.org/3/reference/import.html#open-issues (PR)
It would be really nice to have a diagram.
(import_machinery.rst) how about a section devoted just to the attributes of modules and packages, perhaps expanding upon or supplanting the related entries in the data model reference page?
runpy, pkgutil, et al in the library manual should all get “See Also” links at the top pointing to the new import system section.
Add more explanation regarding the different ways in which __main__ is initialized?
Add more info on __main__ quirks/pitfalls (i.e. copy from PEP 395).

Missing deprecations

Here's a list of related APIs that have not yet been deprecated, but should be. We'll deprecate them in 3.12 and remove them in 3.14.

pkgutil.find_loader()
pkgutil.get_loader() (or it needs to be reimplemented to not use find_loader())

Other:

Do we need to define importlib.abc.PathEntryFinder.find_spec() #103999

PR: gh-97850: Deprecate find_loader and get_loader in pkgutil #98520

PR: GH-97850: Suppress cross-references to the removed module_repr method #104133

PR: GH-97850: Suppress cross-references to removed importlib.util functions #104134

PR: gh-97850: Note in py312 whatsnew that importlib.util.set_loader and importlib.util.module_for_loader have been removed #108719

PR: [3.12] gh-97850: Note in py312 whatsnew that importlib.util.set_loader and importlib.util.module_for_loader have been removed (GH-108719) #108723

PR: gh-97850: remove find_loader and get_loader from pkgutil #119656

PR: gh-97850: Suggest TraversableResources as the alternative for ResourceLoader. #128601

PR: gh-97850: Remove the mention of removal from ResourceReader docs #128602

PR: [3.13] gh-97850: Remove the mention of removal from ResourceReader docs (GH-128602) #128654

PR: [3.12] gh-97850: Remove the mention of removal from ResourceReader docs (GH-128602) #128655

The text was updated successfully, but these errors were encountered:

sobolevn · 2022-10-05T08:43:40Z

I've removed importlib.utils.set_loader, importlib.utils.set_package, and
importlib.utils.module_for_loader from importlib.utils in #97898 🎉

Remove all known instances of module_repr()

Tracking in python#97850 instead.

* main: (66 commits) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) pythongh-93357: Port test cases to IsolatedAsyncioTestCase, part 2 (python#97896) pythongh-95196: Disable incorrect pickling of the C implemented classmethod descriptors (pythonGH-96383) pythongh-97758: Fix a crash in getpath_joinpath() called without arguments (pythonGH-97759) pythongh-74696: Pass root_dir to custom archivers which support it (pythonGH-94251) pythongh-97661: Improve accuracy of sqlite3.Cursor.fetchone docs (python#97662) pythongh-87092: bring compiler code closer to a preprocessing-opt-assembler organisation (pythonGH-97644) pythonGH-96704: Add {Task,Handle}.get_context(), use it in call_exception_handler() (python#96756) pythongh-93738: Documentation C syntax (:c:type:`PyTypeObject*` -> :c:expr:`PyTypeObject*`) (python#97778) pythongh-97825: fix AttributeError when calling subprocess.check_output(input=None) with encoding or errors args (python#97826) Add re.VERBOSE flag documentation example (python#97678) ...

…97935) Remove the open issues section from the import reference Tracking in #97850 instead.

…nce (pythonGH-97935) Remove the open issues section from the import reference Tracking in python#97850 instead. (cherry picked from commit f8edc6f) Co-authored-by: Brett Cannon <[email protected]>

…H-97935) Remove the open issues section from the import reference Tracking in #97850 instead. (cherry picked from commit f8edc6f) Co-authored-by: Brett Cannon <[email protected]>

…ence (GH-97935) (GH-97994) Remove the open issues section from the import reference Tracking in #97850 instead. (cherry picked from commit f8edc6f) Co-authored-by: Brett Cannon <[email protected]> Automerge-Triggered-By: GH:brettcannon

* main: fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965) pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990) pythongh-97850: Remove the open issues section from the import reference (python#97935) Docs: pin sphinx-lint (pythonGH-97992) pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591) Add Pynche's move to the What's new in 3.11 (python#97974) pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785) pythongh-86482: Document assignment expression need for ()s (python#23291) pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949) pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)

* gh-97850: Remove deprecated functions from `importlib.utils` * Rebase and remove `set_package` from diff

* main: (53 commits) pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926) pythonGH-88050: fix race in closing subprocess pipe in asyncio (python#97951) pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962) pythongh-95986: Fix the example using match keyword (python#95989) pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944) pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929) pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610) pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703) pythonGH-88968: Add notes about socket ownership transfers (python#97936) pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528) pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879) docs(typing): add "see PEP 675" to LiteralString (python#97926) pythongh-97850: Remove all known instances of module_repr() (python#97876) I changed my surname early this year (python#96671) pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768) pythongh-91539: improve performance of get_proxies_environment (python#91566) build(deps): bump actions/stale from 5 to 6 (python#97701) pythonGH-95172 Make the same version `versionadded` oneline (python#95172) pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073) pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774) ...

* main: pythonGH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (pythonGH-97996) pythongh-97973: Return all necessary information from the tokenizer (pythonGH-97984) fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965) pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990) pythongh-97850: Remove the open issues section from the import reference (python#97935) Docs: pin sphinx-lint (pythonGH-97992) pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591) Add Pynche's move to the What's new in 3.11 (python#97974) pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785) pythongh-86482: Document assignment expression need for ()s (python#23291) pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)

* main: pythongh-86298: Ensure that __loader__ and __spec__.loader agree in warnings.warn_explicit() (pythonGH-97803) pythongh-82874: Convert remaining importlib format uses to f-str. (python#98005) Docs: Fix backtick errors found by sphinx-lint (python#97998) pythongh-97850: Remove deprecated functions from `importlib.utils` (python#97898) Remove extra spaces in custom openSSL documentation. (python#93568) pythonGH-90985: Revert "Deprecate passing a message into cancel()" (python#97999)

…7876) Remove all known instances of module_repr()

…nce (python#97935) Remove the open issues section from the import reference Tracking in python#97850 instead.

…ython#97898) * pythongh-97850: Remove deprecated functions from `importlib.utils` * Rebase and remove `set_package` from diff

… `importlib.util.module_for_loader` have been removed (#108719) Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed

…r` and `importlib.util.module_for_loader` have been removed (pythonGH-108719) Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed (cherry picked from commit 013a99a) Co-authored-by: Alex Waygood <[email protected]>

…er` and `importlib.util.module_for_loader` have been removed (GH-108719) (#108723) gh-97850: Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed (GH-108719) Note in py312 whatsnew that `importlib.util.set_loader` and `importlib.util.module_for_loader` have been removed (cherry picked from commit 013a99a) Co-authored-by: Alex Waygood <[email protected]>

hugovk · 2023-11-05T20:29:48Z

Edit 2023-11-05: 36/71 (52%)

We're now over 50% of those in the checklist.

hugovk · 2024-06-07T09:42:10Z

Edit 2024-06-07: 55/71 (78%)

We're now at 78% (fixed or unmaintained/abandoned).

…119656) Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Brett Cannon <[email protected]>

…til`` (python#119656) Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Brett Cannon <[email protected]>

tomasr8 · 2024-12-18T13:19:31Z

Related issue: #121604

Here's a couple more things that might need attention:

(docs) ResourceLoader is deprecated and recommends ResourceReader as an alternative which is itself deprecated. We should skip this extra link and recommend TraversableResources directly.
ResourceReader is deprecated and supposed to be removed in 3.14 but does not raise DeprecationWarning - I can add a Deprecation warning in gh-121604: Make sure all deprecated items in importlib raise DeprecationWarning #128007 and we should probably update the docs to say that it'll be removed in 3.16 instead.
Loader.load_module is deprecated with the deprecation stating it will be removed in 3.12. Should we try to remove this method in 3.14? If not, we should update the deprecation message.

In general, lots of methods/attributes are marked as deprecated in the docs for a long time (for example: DEBUG_BYTECODE_SUFFIXES is deprecated since 3.5). It's probably not a good idea to remove them now (especially those that don't yet raise a DeprecationWarning), but we could mark them as deprecated-removed instead, planning to remove them in 3.16?

Would like to know your thoughts on what to do about these :)

brettcannon · 2024-12-18T17:54:32Z

(docs) ResourceLoader is deprecated and recommends ResourceReader as an alternative which is itself deprecated. We should skip this extra link and recommend TraversableResources directly.

👍

ResourceReader is deprecated and supposed to be removed in 3.14 but does not raise DeprecationWarning - I can add a Deprecation warning in gh-121604: Make sure all deprecated items in importlib raise DeprecationWarning #128007 and we should probably update the docs to say that it'll be removed in 3.16 instead.

That's a @jaraco question.

Loader.load_module is deprecated with the deprecation stating it will be removed in 3.12. Should we try to remove this method in 3.14? If not, we should update the deprecation message.

If it has been documented as deprecated for 5 releases and raising an exception for as long, then it can come out.

but we could mark them as deprecated-removed instead, planning to remove them in 3.16?

We should keep it all around for 5 releases of being documented as deprecated as well as raising an exception. There's no rush here as we are just trying to nudge people to better APIs long-term more than avoiding buggy code that's blocking some other change.

jaraco · 2024-12-24T15:30:49Z

ResourceReader is deprecated and supposed to be removed in 3.14 but does not raise DeprecationWarning - I can add a Deprecation warning in gh-121604: Make sure all deprecated items in importlib raise DeprecationWarning #128007 and we should probably update the docs to say that it'll be removed in 3.16 instead.

That's a @jaraco question.

Thanks for the tag.

I'm not certain ResourceReader is even deprecated.

It was in #93963 that importlib.abc.ResourceReader (note not importlib.resources.abc) was deprecated (and subsequently removed). But the deprecation notices were added to the docs for importlib.resources.abc.

It seems this discontinuity has led to previous confusion (e.g. #107352), though it doesn't seem the discontinuity was resolved.

I'm not sure if there was ever an intention to deprecate ResourceReader. It does seem to make sense to do so, given that TraversableResources is to be preferred. Since we've had the deprecation notice in the docs for a while, and if we extend the deadline to Python 3.16, that should provide adequate notice of a removal, though we may choose to extend that deadline.

On the other hand, maybe it makes sense to simply change the documentation to reflect the deprecation without any scheduled removal. That would be my preference. Do you have an opinion @encukou?

tomasr8 · 2024-12-31T12:33:34Z

Thanks to both for your answers!

@jaraco If I understand correctly, you lean more towards keeping ResourceReader deprecated? If that is the case, should we add a DeprecationWarning as well? Currently, the deprecation is only mentioned in the docs. As Brett says, we can also extend the deprecation timeline in the docs to 3.14 + 5.

encukou · 2025-01-02T10:00:15Z

maybe it makes sense to simply change the documentation to reflect the deprecation without any scheduled removal. That would be my preference.

That sounds good to me too. It's no longer the best way to do things, but I don't think that's a good reason to break people's working code -- or even bother them with warnings.

I know some people stick to __file__ simply because it's worked the same way for decades, warts and all. IMO, importlib.resources should be similarly boring. But it won't get there if it nudges users to rewrite old code when it adds new features.

tomasr8 · 2025-01-03T13:14:37Z

Ok I can remove the scheduled removal from the docs of ResourceReader 👍

@encukou since you mentioned not bothering people with warnings, what's your opinion on this issue (and the linked PR): #121604 Should all the mentioned items also raise deprecation warnings?

jaraco · 2025-01-04T08:46:27Z

@jaraco If I understand correctly, you lean more towards keeping ResourceReader deprecated? If that is the case, should we add a DeprecationWarning as well? Currently, the deprecation is only mentioned in the docs. As Brett says, we can also extend the deprecation timeline in the docs to 3.14 + 5.

I do, but I'm not confident there's a clear path to deprecate, because although the interface is deprecated, it's an interface that's used both by providers and consumers of that interface. Even its replacement subclasses that interface, so it's not even obvious to me that a deprecation warning is possible. Sounds like it's best to just leave it documented. Thanks.

encukou · 2025-01-06T10:09:11Z

what's your opinion on this issue (and the linked PR): #121604

My first impression: The deprecation code seems like a bigger maintenance burden than the deprecated API itself; better to just leave it alone. Let's deprecate things that are dangerous or misleading, not just old and limited.
But, I don't think my personal opinion should carry much weight here.

warsaw · 2025-01-07T21:57:19Z

Let's deprecate things that are dangerous or misleading
... or unmaintainable.

If it doesn't match any of those criteria then agreed there's no urgency to remove them and break code.

tomasr8 · 2025-01-07T23:32:19Z

Thanks everyone for your comments! :)

I created #128601 and #128602 to update the deprecations in the docs.

…128602) Remove the mention of removal from ResourceReader docs

…ocs (pythonGH-128602) Remove the mention of removal from ResourceReader docs (cherry picked from commit a1284e9) Co-authored-by: Tomas R. <[email protected]>

…til`` (python#119656) Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Brett Cannon <[email protected]>

warsaw changed the title ~~Task list for cleaning up import system cruft for Python 3.12~~ Meta issue for cleaning up import system cruft Oct 4, 2022

bedevere-bot mentioned this issue Oct 5, 2022

gh-97850: Remove all known instances of module_repr() #97876

Merged

warsaw added the topic-importlib label Oct 5, 2022

warsaw self-assigned this Oct 5, 2022

sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 5, 2022

pythongh-97850: Remove deprecated functions from importlib.utils

cb4e97c

bedevere-bot mentioned this issue Oct 5, 2022

gh-97850: Remove deprecated functions from importlib.utils #97898

Merged

warsaw added a commit that referenced this issue Oct 5, 2022

gh-97850: Remove all known instances of module_repr() (#97876)

5dc3599

Remove all known instances of module_repr()

brettcannon self-assigned this Oct 5, 2022

brettcannon added a commit to brettcannon/cpython that referenced this issue Oct 5, 2022

Remove the open issues section from the import reference

0591f9f

Tracking in python#97850 instead.

brettcannon mentioned this issue Oct 5, 2022

gh-97850: Remove the open issues section from the import reference #97935

Merged

sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 5, 2022

pythongh-97850: Remove deprecated functions from importlib.utils

b3a8ee6

miss-islington mentioned this issue Oct 6, 2022

[3.11] gh-97850: Remove the open issues section from the import reference (GH-97935) #97994

Merged

brettcannon added a commit that referenced this issue Oct 6, 2022

gh-97850: Remove the open issues section from the import reference (#…

f8edc6f

…97935) Remove the open issues section from the import reference Tracking in #97850 instead.

miss-islington mentioned this issue Oct 6, 2022

[3.10] gh-97850: Remove the open issues section from the import reference (GH-97935) #97995

Merged

warsaw pushed a commit that referenced this issue Oct 7, 2022

gh-97850: Remove deprecated functions from importlib.utils (#97898)

27025e1

* gh-97850: Remove deprecated functions from `importlib.utils` * Rebase and remove `set_package` from diff

warsaw mentioned this issue Oct 7, 2022

Remove more deprecated importlib APIs from Python 3.12 #98040

Closed

6 tasks

mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022

pythongh-97850: Remove all known instances of module_repr() (python#9…

0aca8ca

…7876) Remove all known instances of module_repr()

mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022

pythongh-97850: Remove the open issues section from the import refere…

9ffbad4

…nce (python#97935) Remove the open issues section from the import reference Tracking in python#97850 instead.

mpage pushed a commit to mpage/cpython that referenced this issue Oct 11, 2022

pythongh-97850: Remove deprecated functions from importlib.utils (p…

04a52de

…ython#97898) * pythongh-97850: Remove deprecated functions from `importlib.utils` * Rebase and remove `set_package` from diff

sobolevn added a commit to sobolevn/cpython that referenced this issue Oct 21, 2022

pythongh-97850: Deprecate find_module and get_module in pkgutil

b7de6df

bedevere-bot mentioned this issue Aug 31, 2023

[3.12] gh-97850: Note in py312 whatsnew that importlib.util.set_loader and importlib.util.module_for_loader have been removed (GH-108719) #108723

Merged

wallrite mentioned this issue Apr 18, 2024

Python 3.12: AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? ronaldoussoren/altgraph#23

Closed

picnixz mentioned this issue May 28, 2024

gh-97850: remove find_loader and get_loader from pkgutil #119656

Merged

ferdnyc mentioned this issue Jun 16, 2024

Python version update wireviz/WireViz#309

Open

cHiv0rz mentioned this issue Oct 18, 2024

gsutil 5.31 doesn't work with Python 3.12.7 in alpine GoogleCloudPlatform/gsutil#1808

Closed

hugovk added a commit that referenced this issue Nov 1, 2024

gh-97850: remove find_loader and get_loader from pkgutil (#…

464a7a9

…119656) Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Brett Cannon <[email protected]>

picnixz added a commit to picnixz/cpython that referenced this issue Dec 8, 2024

pythongh-97850: remove find_loader and get_loader from ``pkgu…

21d2e70

…til`` (python#119656) Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Brett Cannon <[email protected]>

This was referenced Jan 7, 2025

gh-97850: Suggest TraversableResources as the alternative for ResourceLoader. #128601

Open

gh-97850: Remove the mention of removal from ResourceReader docs #128602

Merged

jaraco pushed a commit that referenced this issue Jan 8, 2025

gh-97850: Remove the mention of removal from ResourceReader docs (#…

a1284e9

…128602) Remove the mention of removal from ResourceReader docs

This was referenced Jan 8, 2025

[3.13] gh-97850: Remove the mention of removal from ResourceReader docs (GH-128602) #128654

Open

[3.12] gh-97850: Remove the mention of removal from ResourceReader docs (GH-128602) #128655

Open

ebonnal pushed a commit to ebonnal/cpython that referenced this issue Jan 12, 2025

pythongh-97850: remove find_loader and get_loader from ``pkgu…

eaf9ef4

…til`` (python#119656) Co-authored-by: Hugo van Kemenade <[email protected]> Co-authored-by: Brett Cannon <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meta issue for cleaning up import system cruft #97850

Meta issue for cleaning up import system cruft #97850

warsaw commented Oct 4, 2022 •

edited by bedevere-app bot

Loading

sobolevn commented Oct 5, 2022

hugovk commented Nov 5, 2023

hugovk commented Jun 7, 2024

tomasr8 commented Dec 18, 2024

brettcannon commented Dec 18, 2024

jaraco commented Dec 24, 2024

tomasr8 commented Dec 31, 2024

encukou commented Jan 2, 2025

tomasr8 commented Jan 3, 2025

jaraco commented Jan 4, 2025

encukou commented Jan 6, 2025

warsaw commented Jan 7, 2025

tomasr8 commented Jan 7, 2025

Meta issue for cleaning up import system cruft #97850

Meta issue for cleaning up import system cruft #97850

Comments

warsaw commented Oct 4, 2022 • edited by bedevere-app bot Loading

Removing deprecated APIs

Update docs

Missing deprecations

sobolevn commented Oct 5, 2022

hugovk commented Nov 5, 2023

hugovk commented Jun 7, 2024

tomasr8 commented Dec 18, 2024

brettcannon commented Dec 18, 2024

jaraco commented Dec 24, 2024

tomasr8 commented Dec 31, 2024

encukou commented Jan 2, 2025

tomasr8 commented Jan 3, 2025

jaraco commented Jan 4, 2025

encukou commented Jan 6, 2025

warsaw commented Jan 7, 2025

tomasr8 commented Jan 7, 2025

warsaw commented Oct 4, 2022 •

edited by bedevere-app bot

Loading