Teach FawltyDeps to automatically discover Python environments inside the project #326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jherland
force-pushed
the
jherland/prepare-for-traversing-pyenvs
branch
from
626aa6c to
b3d8bef
Compare
jherland
force-pushed
the
jherland/traverse-pyenvs
branch
2 times, most recently
from
f8b8261 to
b6aa14e
Compare
6 tasks
jherland
force-pushed
the
jherland/prepare-for-traversing-pyenvs
branch
from
b3d8bef to
598c293
Compare
jherland
force-pushed
the
jherland/traverse-pyenvs
branch
from
b6aa14e to
935cf82
Compare
jherland
force-pushed
the
jherland/prepare-for-traversing-pyenvs
branch
from
598c293 to
36a78ab
Compare
jherland
force-pushed
the
jherland/traverse-pyenvs
branch
from
935cf82 to
6f455b1
Compare
|
🎉 All dependencies have been resolved ! |
jherland
force-pushed
the
jherland/traverse-pyenvs
branch
2 times, most recently
from
76d61c8 to
12fadc0
Compare
mknorps
reviewed
Jul 4, 2023
There was a problem hiding this comment.
As usual Johan, your PRs are elegant, well written and are a great contribution to the project <3
I am posting a comment, not approval for now, as I have to go through tests and documentation. The comments are mostly nits or me making sure that I follow your line of thoughts :)
Nour-Mws
reviewed
Jul 4, 2023
mknorps
approved these changes
Jul 11, 2023
In addition to finding/creating CodeSource and DepsSource objects in traverse_project.find_sources(), we now also add support for PyEnvSource objects. Each of these objects encapsulate a single directory within a Python environment where we expect to find Python packages installed. In addition to validating the settings.pyenvs paths (similar to what the LocalPackageResolver constructor already does), this also adds the logic to find new Python environments within the directories being traversed. That is, until now we have only supported entries in settings.pyenvs pointing directly to (a directory within) a Python environment, but with this commit we can also start finding Python environments beneath a given directory. In others words: given a project containing a ".venv" subdir, we previously needed 'pyenvs' to point directly at ".venv" in order to find this Python environment, but with this commit we can soon support 'pyenvs' pointing at the project root and still be able to automatically find ".venv" inside the project. Nothing of this is user-visible yet, as we still require the high-level business logic in fawltydeps.main to actually request, and use the new PyEnvSource objects created in this commit. Based on the recent additions to test_sample_projects and sample_projects, we add a bunch of tests to verify that we are able to both precisely specify which Python environments should be included in our analysis, as well as being able to automatically find and use many different Python environments given a more general specification.
This starts requesting PyEnvSource objects when we call traverse_project.find_sources(), and pass the paths from these object on to resolve_dependencies(). In other words, this commit is when we first start actually using the new traversal logic for Python environments in FawltyDeps. On the test side, we can finally remove the overly explicit specification needed to correctly traverse/parse the pyenv_galore sample project. Instead, this project is now successfully handled with default settings: - pyenvs set to the project root is able to auto-detect all the Python environments found within. - As a result, there is no code, nor any deps files found (since the project consists of Python environments only).
Update README.md to reflect that we're now using the _current_ Python environment (i.e. 'sys.path') only as an ultimate fallback when (a) `--pyenv` is not used, and (b) we're unable to find Python environments under 'basepath'. Update the related FAQs accordingly. Also update the --help text to reflect that `basepath` now also affect Python environment calculation.
Add descriptions of each instance member in our new *Source classes. Suggested-by: Maria Knorps <[email protected]>
The --code/--deps/--pyenv options was documented as defaulting to the current directory (which is only true when basepaths is not used), and basepaths was NOT documented as defaulting to the current directory. The correct way to document this (which reflects the actual logic in the code) is that --code/--deps/--pyenv all default to basepaths when not given, and that basepaths default to the current directory when not given. With this change, a casual reader might have to follow the reference from --code/--deps/--pyenv to basepaths to discover that the _ultimate_ default for these options is indeed the current directory. But at least the relationships involving basepaths are now _correctly_ reflected in out documentation. Suggested-by: Maria Knorps <[email protected]>
This was referenced Jul 14, 2023
Suggested-by: Nour El Mawass <[email protected]>
jherland
force-pushed
the
jherland/traverse-pyenvs
branch
from
488cf80 to
92f3f0d
Compare
jherland
added a commit
that referenced
this pull request
Jul 28, 2023
This is a rather embarrasing omission from PR #326 (Teach FawltyDeps to automatically discover Python environments inside the project): Although that PR did include code to traverse within a given --pyenv path to find Python environments, the PR "forgot" to actually switch the _default_ behavior of --pyenv: The default value of settings.pyenvs remained an empty set, and there were no changes to have basepath influence the value of settings.pyenvs. This commit fixes that: - When neither --pyenv nor basepath is given, settings.pyenvs should default to the current directory. - When --pyenv is not given, but basepath is given, basepath should override the default settings.pyenvs. - When --pyenv is given, it overrides basepath. In short settings.pyenvs should behave exactly like .code and .deps.
jherland
added a commit
that referenced
this pull request
Jul 31, 2023
This is a rather embarrasing omission from PR #326 (Teach FawltyDeps to automatically discover Python environments inside the project): Although that PR did include code to traverse within a given --pyenv path to find Python environments, the PR "forgot" to actually switch the _default_ behavior of --pyenv: The default value of settings.pyenvs remained an empty set, and there were no changes to have basepath influence the value of settings.pyenvs. This commit fixes that: - When neither --pyenv nor basepath is given, settings.pyenvs should default to the current directory. - When --pyenv is not given, but basepath is given, basepath should override the default settings.pyenvs. - When --pyenv is given, it overrides basepath. In short settings.pyenvs should behave exactly like .code and .deps.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(Depends on #325)
This is at the heart of solving #249, and IMHO one of the major pieces remaining in the Mapping Strategy milestone.
With this, FawltyDeps is able to:
basepath).codeand.depsCommits:
traverse_project: Add support for finding/validating Python environmentsmain: UsePyEnvSourceobjects when resolving dependencies