-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Wait for request stream to flush before returning resolution #2374
Conversation
@@ -197,42 +196,40 @@ impl<'a, Provider: ResolverProvider> Resolver<'a, Provider> { | |||
let requests_fut = self.fetch(request_stream).fuse(); | |||
|
|||
// Run the solver. | |||
let resolve_fut = self.solve(&request_sink).fuse(); | |||
let resolve_fut = self.solve(request_sink).fuse(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Taking ownership means that request_sink
is dropped as soon as the future completes.
No meaningful change, as expected:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice
@@ -27,7 +27,7 @@ pub enum ResolveError { | |||
#[error(transparent)] | |||
Client(#[from] uv_client::Error), | |||
|
|||
#[error("The channel is closed, was there a panic?")] | |||
#[error("The channel closed unexpectedly")] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added the panic message to all the broken channel cases because in my experience this error happens when there was a panic in another thread, so the error message you get is ~irrelevant, while scrolling up another thread spilled a first panic to stderr that has the actually relevant failure
I also benchmarked on home-assistant to be safe (this branch is "faster" but I think it's just noise):
|
Summary
This is a more robust fix for #2300.
The basic issue is:
--no-deps
, but the pre-fetch was causing us to attempt to build a package to get its dependencies. The resolution would then finish before the build completed.)Index
will be marked as "waiting" for that response -- but it'll never come through.Index
, to see if we should fetch or are waiting for that response, we'll end up waiting for it forever, since it looks like it's in-flight (but isn't). (In the linked issue, we had to build the source distribution for the install phase ofpip install
, butsetuptools
was in this bad state from the resolve phase.)This PR modifies the resolver to ensure that we flush the stream of requests before returning. Specifically, we now
join
rather thanselect
between the resolution and request-handling futures.This could be wasteful, since we don't need those requests, but it at least ensures that every
.wait
is followed by.done
. In practice, I expect this not to have any significant effect on performance, since we end up using the pre-fetched distributions almost every time.Test Plan
I ran through the test plan from #2373, but ran the build 10 times and ensured it never crashed. (I reverted #2373, since that also fixes the issue in the proximate case, by never fetching
setuptools
during the resolve phase.)I also added logging to verify that requests are being handled after the resolution completes, as expected.
I also introduced an arbitrary error in
fetch
to ensure that the error was immediately propagated.