Implement pkg_resources provider

[indygreg]

It looks like pkg_resources is extensible and we can implement our own provider for teaching it to use PyOxidizer's custom loader. A custom provider may not be needed for all functionality. But it will almost certainly be needed for some functionality (e.g. importlib.metadata).

See https://github.com/pypa/setuptools/blob/master/pkg_resources/__init__.py for the source code.

[mharbison72]

Would this prevent importing pkg_resources with Mercurial? I see a commit[1] saying import pkg_resources works, so I'm guessing no, unless there was a regression.

What I'd like to do is to grab version info from the keyring and mercurial_keyring packages to display with hg version -v. The keyring package used to do this to implement the __version__ attribute[2], but dropped that a while ago due to overhead.[3] The overhead seems fine in a getversion() function on the extension, and seems to work with py2exe, wrapper.exe compiled for either py2 or py3, and plain py -3 hg version -v. But it doesn't make sense to implement it if it can't work with pyoxidizer. I'm open to other ideas on how to get this version info too, but I doubt either maintainer will want to manually update a version field in code.

$ "c:\Users\Matt\hg\build\pyoxidizer\x86_64-pc-windows-msvc\release\app_windows/hg.exe" --config extensions.mercurial_keyring=mercurial_keyring_version_test/mercurial_keyring.py version
 -v
*** failed to import extension evolve: No module named 'evolve'
*** failed to import extension topic: No module named 'topic'
Traceback (most recent call last):
  File "c:/Users/Matt/projects/mercurial_keyring_version_test/mercurial_keyring.py", line 82, in getversion
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
Mercurial Distributed SCM (version 5.7+hg155.c3c7a86e9c24)
(see https://mercurial-scm.org for more information)

[1] 58d79ef
[2] https://github.com/jaraco/keyring/blob/9066fbf0abee465bbe230e0e76d2598a9f08569c/keyring/__init__.py#L9
[3] https://github.com/jaraco/keyring/blob/main/CHANGES.rst#1100

[indygreg]

Interesting bug report.

I'd like to think that import pkg_resources should just work from a PyOxidizer built binary, as setuptools + pkg_resources should be in the packaged standard library. And I can confirm that the default application from pyoxidizer init-config-file + pyoxidizer run does work on Windows. However, hg.exe --debugger fails to import pkg_resources.

If I run sys.meta_path[0].find_spec("pkg_resources", None) from Mercurial's debugger, that returns None. So the pkg_resources module isn't getting indexed by PyOxidizer. Why, I'm not sure. But this smells like a bug.

[mharbison72]

Not sure if this is related, but while it works from the main python install and --config extensions=/path/to/ext.py, I get a weird demandimport related error in a venv that goes away with HGDEMANDIMPORT=disable:

(venv-thg-py3.8.1-x64) C:\Users\Matt\projects\thg_py3>hg version -v --pager=no
Traceback (most recent call last):
  File "c:\Users\Matt\projects\venv-thg-py3.8.1-x64\lib\site-packages\mercurial_keyring.py", line 895, in _getversion
    dist = pkg_resources.get_distribution(mod)
  File "C:\Program Files\Python38\lib\importlib\util.py", line 245, in __getattribute__
    self.__spec__.loader.exec_module(self)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\Users\Matt\projects\venv-thg-py3.8.1-x64\lib\site-packages\pkg_resources\__init__.py", line 81, in <module>
    from pkg_resources.extern import packaging
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 628, in _load_backward_compatible
  File "C:\Program Files\Python38\lib\importlib\util.py", line 250, in __getattribute__
    raise ValueError(f"module object for {original_name!r} "
ValueError: module object for 'pkg_resources._vendor.packaging' substituted in sys.modules during a lazy load
Traceback (most recent call last):
  File "c:\Users\Matt\projects\venv-thg-py3.8.1-x64\lib\site-packages\mercurial_keyring.py", line 895, in _getversion
    dist = pkg_resources.get_distribution(mod)
AttributeError: module 'pkg_resources' has no attribute 'get_distribution'
Mercurial Distributed SCM (version 5.6.1)
(see https://mercurial-scm.org for more information)

No idea if that helps narrow down the problem.