Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search for all python3.x in PATH #5148

Merged
merged 3 commits into from
Jul 18, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ unscanny = { version = "0.1.0" }
url = { version = "2.5.0" }
urlencoding = { version = "2.1.3" }
walkdir = { version = "2.5.0" }
which = { version = "6.0.0" }
which = { version = "6.0.0", features = ["regex"] }
winapi = { version = "0.3.9", features = ["fileapi", "handleapi", "ioapiset", "winbase", "winioctl", "winnt"] }
winreg = { version = "0.52.0" }
wiremock = { version = "0.6.0" }
Expand Down
85 changes: 74 additions & 11 deletions crates/uv-python/src/discovery.rs
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
use itertools::{Either, Itertools};
use regex::Regex;
use same_file::is_same_file;
use std::borrow::Cow;
use std::env::consts::EXE_SUFFIX;
use std::fmt::{self, Formatter};
use std::{env, io};
use std::{env, io, iter};
use std::{path::Path, path::PathBuf, str::FromStr};

use itertools::Itertools;
use pep440_rs::{Version, VersionSpecifiers};
use same_file::is_same_file;
use thiserror::Error;
use tracing::{debug, instrument, trace};
use which::{which, which_all};

use pep440_rs::{Version, VersionSpecifiers};
use uv_cache::Cache;
use uv_configuration::PreviewMode;
use uv_fs::Simplified;
Expand Down Expand Up @@ -360,18 +361,17 @@ fn python_executables_from_search_path<'a>(
let search_path =
env::var_os("UV_TEST_PYTHON_PATH").unwrap_or(env::var_os("PATH").unwrap_or_default());

let possible_names: Vec<_> = version
.unwrap_or(&VersionRequest::Any)
.possible_names(implementation)
.collect();
let version_request = version.unwrap_or(&VersionRequest::Any);
let possible_names: Vec<_> = version_request.possible_names(implementation).collect();

trace!(
"Searching PATH for executables: {}",
possible_names.join(", ")
);

// Split and iterate over the paths instead of using `which_all` so we can
// check multiple names per directory while respecting the search path order
// check multiple names per directory while respecting the search path order and python names
// precedence.
let search_dirs: Vec<_> = env::split_paths(&search_path).collect();
search_dirs
.into_iter()
Expand All @@ -391,8 +391,11 @@ fn python_executables_from_search_path<'a>(
which::which_in_global(&*name, Some(&dir))
.into_iter()
.flatten()
// We have to collect since `which` requires that the regex outlives its
// parameters, and the dir is local while we return the iterator.
.collect::<Vec<_>>()
})
.chain(find_all_minor(implementation, version_request, &dir_clone))
.filter(|path| !is_windows_store_shim(path))
.inspect(|path| trace!("Found possible Python executable: {}", path.display()))
.chain(
Expand All @@ -410,6 +413,65 @@ fn python_executables_from_search_path<'a>(
})
}

/// Find all acceptable `python3.x` minor versions.
///
/// For example, let's say `python` and `python3` are Python 3.10. When a user requests `>= 3.11`,
/// we still need to find a `python3.12` in PATH.
fn find_all_minor(
implementation: Option<&ImplementationName>,
version_request: &VersionRequest,
dir_clone: &Path,
) -> impl Iterator<Item = PathBuf> {
match version_request {
VersionRequest::Any | VersionRequest::Major(_) | VersionRequest::Range(_) => {
let regex = if let Some(implementation) = implementation {
Regex::new(&format!(
r"^({}|python3)\.\d\d?{}$",
regex::escape(&implementation.to_string()),
regex::escape(EXE_SUFFIX)
))
.unwrap()
} else {
Regex::new(&format!(r"^python3\.\d\d?{}$", regex::escape(EXE_SUFFIX))).unwrap()
};
let all_minors = which::which_re_in(&regex, Some(&dir_clone))
.into_iter()
.flatten()
.filter(move |path| {
// Filter out interpreter we already know have a too low minor version.
let minor = path
.file_name()
.and_then(|filename| filename.to_str())
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's silly we can't use capture groups for this but the which api is too locked down for that

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we should just not use which and we should iterate over the executables in the directory manually. A regex feels heavier than it needs to be. Wdyt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's an implementation i used to have (without which it's much better than the current one), but it failed because which doesn't expose Checker so we'd have to vendor that code from them to check whether we can use a specific file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with vendoring that code if you are.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added is_executable

.and_then(|filename| {
if EXE_SUFFIX.is_empty() {
Some(filename)
} else {
filename.strip_suffix(EXE_SUFFIX)
}
})
.and_then(|stem| stem.strip_prefix("python3."))
.and_then(|minor| minor.parse().ok());
if let Some(minor) = minor {
// Optimization: Skip generally unsupported Python versions without querying.
if minor < 7 {
return false;
}
// Optimization 2: Skip excluded Python (minor) versions without querying.
if !version_request.matches_major_minor(3, minor) {
return false;
}
}
true
})
.collect::<Vec<_>>();
Either::Left(all_minors.into_iter())
}
VersionRequest::MajorMinor(_, _) | VersionRequest::MajorMinorPatch(_, _, _) => {
Either::Right(iter::empty())
}
}
}

/// Lazily iterate over all discoverable Python interpreters.
///
/// Note interpreters may be excluded by the given [`EnvironmentPreference`] and [`PythonPreference`].
Expand Down Expand Up @@ -1596,12 +1658,13 @@ mod tests {
use assert_fs::{prelude::*, TempDir};
use test_log::test;

use super::Error;
use crate::{
discovery::{PythonRequest, VersionRequest},
implementation::ImplementationName,
};

use super::Error;

#[test]
fn interpreter_request_from_str() {
assert_eq!(PythonRequest::parse("any"), PythonRequest::Any);
Expand Down
26 changes: 26 additions & 0 deletions crates/uv-python/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1752,6 +1752,32 @@ mod tests {
Ok(())
}

#[test]
fn find_python_all_minors() -> Result<()> {
let mut context = TestContext::new()?;
context.add_python_interpreters(&[
(true, ImplementationName::CPython, "python", "3.10.0"),
(true, ImplementationName::CPython, "python3", "3.10.0"),
(true, ImplementationName::CPython, "python3.12", "3.12.0"),
])?;

let python = context.run(|| {
find_python_installation(
&PythonRequest::parse(">= 3.11"),
EnvironmentPreference::Any,
PythonPreference::OnlySystem,
&context.cache,
)
})??;
assert_eq!(
python.interpreter().python_full_version().to_string(),
"3.12.0",
"We should find matching minor version even if they aren't called `python` or `python3`"
);

Ok(())
}

#[test]
fn find_python_pypy_prefers_executable_with_implementation_name() -> Result<()> {
let mut context = TestContext::new()?;
Expand Down
Loading