Revert "Revert "Adapt scheduler to work with dynamic derivations

[Ericson2314]

Motivation

This fixes dynamic derivations, reverting #9081.
#9052 However will be reintroduced unless this is modified somehow.

Context

This reverts commit 8440afb.

Priorities

Add 👍 to pull requests you find important.

[roberth]

This might be a tall order, but could this change be split?
That'd be one way to make progress and narrow down where the problem really is.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/pre-rfc-implement-dependency-retrieval-primitive/43418/9

[roberth]

I believe the changes after #11005 will either uncover or silently fix the bug.

[L-as]

I wouldn't be so sure yet; I haven't rewritten DerivationGoal.

But the subsequent PRs will.

[roberth]

@L-as yeah, I meant after, and it is only a belief.
I didn't mean to set a bar for your work; I just wanted to highlight it because I think it's good, and helpful for delivering dynamic derivations.

[Ericson2314]

Yeah I think it will help too: there is no more need for outer derivation goal with that change, because we don't need separate types just to have different local variables persisted across yield points.

[ConnorBaker]

Hi all — as a quick status update, did #11005 resolve the issue blocking this from proceeding? If not, any ideas on how to proceed?

[L-as]

Pretty sure it hasn't. I'm still doing work in this direction that should fix it naturally.

[Ericson2314]

Yeah looking forward to closing this once @L-as's work obviates it :)