A VMM in state 'starting' can receive requests

[jmpesp]

If a VMM is in state 'starting' and cannot activate a given disk, it will hang. Region replacement or region snapshot replacement may alter the VCR of that disk to one that could be activated, but the current drive saga will not attempt to send requests to a VMM in 'starting'. Fix this - any time a Propolis is expected to be there, it should be ok to receive these requests. Otherwise a repair can be stuck, and the VMM could fail to stop if it gets stuck deactivating too!

[gjcolombo]

Have we verified empirically that region snapshot/replacement work as intended on a Propolis that hasn't yet reached Running? I'm not certain they will (and would need to go look carefully at the relevant Propolis code to check).

[hawkw]

I agree with @gjcolombo --- it's worth making sure that propolis-server can actually handle region replacement requests in that state.

[hawkw]

I believe that a VMM isn't in the Starting state unless the Propolis server process does exist, FWIW --- if memory serves, that's the distinction between Starting and Creating. It may not yet be incarnating an instance, though.

[gjcolombo]

I looked at this a bit. In Propolis there are at least two things we need to consider:

• Sled agent maps Propolis's "Creating" state to Omicron's "Starting" state, so it's possible that the Propolis API call will return a "no instance" error (it hasn't gotten far enough along to believe it has an active VMM yet). This is probably fine; if you can detect this error you can just retry.
• Probably more significantly: Propolis queues VCR change requests to the worker task that's otherwise responsible for initializing VMs and changing their state. (This is so that we don't have to reason about what happens if you try to mutate an instance's configuration while it's migrating.) If a VM is stuck in Starting because it can't activate a Crucible disk, the region replacement that might fix the situation will queue up behind the "start the VM" task and won't actually resolve.

It might be possible to fix the latter issue, but it's probably going to take a fair amount of effort on the Propolis side of the house. (We'd also need to be sure that if we have a Crucible upstairs that's stuck in a Volume::activate, and we call Volume::target_replace on it, then that succeeds and allows the activate call to resolve.)

[gjcolombo]

Filed oxidecomputer/propolis#841 to track the relevant Propolis enhancement.