Refactor lazy imports

[ofek]

Currently, to keep the CLI responsive we import most things where they are used rather than at the top of modules. The scientific Python community has coalesced on a design and a reference implementation that we should adopt:

• https://scientific-python.org/specs/spec-0001/
• https://github.com/scientific-python/lazy_loader

This will be used ubiquitously throughout not just Hatch but also Hatchling.

Special care must be taken when adding dependencies to Hatchling and I think this is fine for the following reasons:

1. The aforementioned package has no dependencies itself
2. That package builds with setuptools so re-distributors will not encounter cyclical dependencies
3. We will be getting rid of the dependency pluggy soon so it will just be a trade

[hoechenberger]

We have adopted this lazy loading approach in MNE-Python and I have to say I'm not a fan. It feels extremely hacky and basically abuses .pyi files to achieve something that's not part of the Python standard library. My personal suggestion is, if you can live with nested imports, use these instead.

The scientific Python community has coalesced on a design

I'd also say this is not quite correct. A few people wrote a proposal, and a handful of packages have adopted it. That's about it.

There are very good reasons why the PEP for lazy loading was rejected not so long ago.

[ofek]

Oh very interesting, thank you!

[hoechenberger]

Don't get me wrong, I'm not telling you what and what not to do :) I just wanted to share my personal feelings and experiences here.

[hugovk]

Upstream at CPython, there have been recent PRs to defer some imports in stdlib modules to improve import time: python/cpython#109653

If you see some other candidates, feel free to comment on the issue. No guarantees though, some maintainers are cautious about these kind of changes.

[ofek]

I was unaware of such an effort, that's awesome! That will help CLIs everywhere in perpetuity.

edit: also apps running in function-based models like AWS Lambda

[djcopley]

Forgive me if this is duplicative information, but python importlib has lazy loading utilities available: https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader

[ofek]

Yes but you have to use that programmatically rather than importing as usual or something that resembles imports.

[Sachaa-Thanasius]

Apologies for the issue resurrection, but I was curious what other options have been considered beyond SPEC 1 and the basic importlib lazy_import recipe, or whether this "refactor" is no longer desirable/planned. If the latter is not the case, a few ideas for how to maybe get something more ergonomic than SPEC 1 come to mind:

• Creating a custom meta-path finder that wraps the loaders of found module specs with importlib.util.LazyLoader, then putting that finder in sys.meta_path temporarily via a context manager. That way, one can place import statements inside context manager blocks to change their behavior. A simple version needs very few lines of code, but it won't be (fully) lazy for a few different types of import statements, e.g. from imports and submodule imports. Still, it's an option. Here's an example implementation. mercurial does something similar with their lazy import hook.

• Using slothy (no external dependencies) or a similar custom version.

• Using defer_imports (no external dependencies) or a similar custom version.

Disclaimer: The last one is mine. Also, you can avoid using any of the linked code by (re)implementing something like it yourself if you want; I'm more so using it as examples of a few different possible solution mechanisms to learn whether any fit the needs of hatch, and if none do, learn what hatch does need that existing lazy import implementations don't fulfill.

[ofek]

It's unlikely I would do something that would modify the interpreter generally because we support plugins and don't want to impact them at all.

[Sachaa-Thanasius]

Modifying the interpreter would be going farther than even I imagined, haha. Makes total sense. My previous concrete suggestions, at least, all have context-manager-level or specific-package-level scoping for affecting the behavior of import statements or have configuration options for tuning what they affect, so they'd be much more conservative.

Anyway, just wanted to float some ideas. If they're too invasive for this use case, so be it.

[ofek]

I appreciate the effort and could be wrong, I just took a cursory glance because I don't have much time right now. I'll circle back to this eventually, thanks!

[Sachaa-Thanasius]

No worries. Creating better forms of lazy importing has been a personal passion as of late, so I figured I'd give my two cents in hopes of helping, that's all.