Move to importlib-metadata for performance improvements

[asottile]

setup

(with ~some amount of pip requirements installed -- I used https://github.com/Yelp/paasta as it installs a bunch of things and I know they all build on my machine)

$ pip freeze -l | wc -l
165

I used "best of 5" when picking the outputs below -- the version 0.0.0 is because I didn't have setuptools-scm when I built the dist -- just ignore that ;)

before

$ time tox --version
3.12.1 imported from /tmp/paasta/.tox/py/lib/python3.6/site-packages/tox/__init__.py

real	0m0.298s
user	0m0.270s
sys	0m0.029s

after

$ time tox --version
0.0.0 imported from /tmp/paasta/.tox/py/lib/python3.6/site-packages/tox/__init__.py

real	0m0.208s
user	0m0.200s
sys	0m0.008s

savings

~90ms -- so somewhere between 30-40%

Contribution checklist:

(also see CONTRIBUTING.rst for details)

• wrote descriptive pull request text
• added/updated test(s)
• updated/extended the documentation
• added relevant issue keyword
  in message body
• added news fragment in changelog folder
  • fragment name: <issue number>.<type>.rst for example (588.bugfix.rst)
  • <type> is must be one of bugfix, feature, deprecation,breaking, doc, misc
  • if PR has no issue: consider creating one first or change it to the PR number after creating the PR
  • "sign" fragment with "by :user:<your username>"
  • please use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files - by :user:superuser."
  • also see examples
• added yourself to CONTRIBUTORS (preserving alphabetical order)

[gaborbernat]

shouldn't we version guard from above?

[asottile]

yessss I wasn't sure what to put here, I'll go with >=0.12,<1

[gaborbernat]

can you explain this check here 🤔 maybe put a comment because reading it seems kinda cryptic; why we need the extra?

[asottile]

yeah this is a little bit weird

it's essentially a reimplement of pkg_resources.Requirement.requires but using the importlib-metadata / packaging primitives

the pkg_resources api has:

    def requires(self, extras=()):

so by default it does not pass extras at all.

the importlib_metadata requires ends up looking like this for tox:

>>> for req in importlib_metadata.distribution('tox').requires:
...     print(f'- {req}')
... 
- setuptools>=30.0.0
- pluggy<1,>=0.3.0
- py<2,>=1.4.17
- six<2,>=1.0.0
- virtualenv>=14.0.0
- toml>=0.9.4
- filelock<4,>=3.0.0
- sphinx<3,>=2.0.0; extra == "docs"
- towncrier>=18.5.0; extra == "docs"
- pygments-github-lexers>=0.0.5; extra == "docs"
- sphinxcontrib-autoprogram>=0.1.5; extra == "docs"
- freezegun<1,>=0.3.11; extra == "testing"
- pathlib2<3,>=2.3.3; extra == "testing"
- pytest<5,>=3.0.0; extra == "testing"
- pytest-cov<3,>=2.5.1; extra == "testing"
- pytest-mock<2,>=1.10.0; extra == "testing"
- pytest-xdist<2,>=1.22.2; extra == "testing"
- pytest-randomly<2,>=1.2.3; extra == "testing"
- flaky<4,>=3.4.0; extra == "testing"
- psutil<6,>=5.6.1; (python_version != "3.4") and extra == "testing"

Converting one of those later ones into packaging primitives:

>>> from packaging.requirements import Requirement
>>> req = Requirement('pytest-randomly<2,>=1.2.3; extra == "testing"')
>>> req.marker
<Marker('extra == "testing"')>
>>> req.marker.evaluate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/tox/venv/lib/python3.6/site-packages/packaging/markers.py", line 296, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/tmp/tox/venv/lib/python3.6/site-packages/packaging/markers.py", line 221, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/tmp/tox/venv/lib/python3.6/site-packages/packaging/markers.py", line 203, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
>>> req.marker.evaluate({'extra': ''})
False
>>> req.marker.evaluate({'extra': 'testing'})
True

[asottile]

oh, heh, I was going to add the bounds for importlib-metadata -- I'll do that in another PR 😆