Manage experiments queue

[dberenbaum]

Following up on the dvc queue commands, users should be able to manage queued experiments in VS Code.

For background, see:

• Epic: exp queue CLI dvc#7592
• ref: document dvc queue dvc.org#3715

Possible changes to VS Code:

• Options to show/hide queued and failed experiments in the experiments table
• Add to the queue while other experiment tasks are running
• Drop rows from the queue
• Kill running experiment tasks
• Gracefully stop the queue (finish currently running experiments but don't start new ones)
• Adjust number of jobs for processing the queue

[mattseddon]

Options to show/hide queued and failed experiments in the experiments table

This links in with the suggestion in #1995 (comment) to split out the contents of dvc queue status into another tree view.

Add to the queue while other experiment tasks are running

Straightforward change

Drop rows from the queue

Users can already do this but we could easily move the action to a separate tree/view associated with managing the queue.

Kill running experiment tasks

👍🏻

Gracefully stop the queue (finish currently running experiments but don't start new ones)

👍🏻

Adjust number of jobs for processing the queue

[Q] Does this mean whilst the queue is being processed or simply when it is being started. The later should be easy. We can use a multi-step quick pick.

Next step

We can definitely do all the above from the command palette. I would also advocate that we add another dvc queue status based tree in the view container and add the same actions there.

Simplest thing is probably for me to make a POC for the new treeView with some basic functionality and we can decide whether or not we want to keep going with it.

@karajan1001 how difficult would it be to add the --show-json or --json flag to dvc queue status and provide me with some json output?

[karajan1001]

Options to show/hide queued and failed experiments in the experiments table

This links in with the suggestion in #1995 (comment) to split out the contents of dvc queue status into another tree view.

Add to the queue while other experiment tasks are running

Straightforward change

Drop rows from the queue

Users can already do this but we could easily move the action to a separate tree/view associated with managing the queue.

Kill running experiment tasks

👍🏻

Gracefully stop the queue (finish currently running experiments but don't start new ones)

👍🏻

Adjust number of jobs for processing the queue

[Q] Does this mean whilst the queue is being processed or simply when it is being started. The later should be easy. We can use a multi-step quick pick.

Next step

We can definitely do all the above from the command palette. I would also advocate that we add another dvc queue status based tree in the view container and add the same actions there.

Simplest thing is probably for me to make a POC for the new treeView with some basic functionality and we can decide whether or not we want to keep going with it.

@karajan1001 how difficult would it be to add the --show-json or --json flag to dvc queue status and provide me with some json output?

If the output json format is well defined. I think it could be don't in half a day.

[dberenbaum]

[Q] Does this mean whilst the queue is being processed or simply when it is being started. The later should be easy. We can use a multi-step quick pick.

It's possible to do while the queue is being processed. Whether that's needed in VS Code probably depends on the level of effort.

[mattseddon]

It's possible to do while the queue is being processed. Whether that's needed in VS Code probably depends on the level of effort.

If you LMK the command I can add that functionality.

[mattseddon]

[Q] Will running dvc exp remove clean up the associated queue task(s) from dvc queue status?

It doesn't seem like this is the case from reading iterative/dvc.org#3715. If we are going to try and hide this concept then I think that is something that should be handled by either the CLI or extension. IMO it should be handled by default. If the user wants to keep the logs around then they can pass a flag.

[mattseddon]

After starting to implement dvc queue kill, dvc queue start & dvc queue stop I realised that we are blocked without an indicator that the queue is currently processing tasks. We need this because we conditionally make these commands available. E.g - If there are no workers we show start, if there is at least one we show kill & stop.

From here & #1995 the only thing that I currently need from dvc queue status is the number of active and idle workers.

@karajan1001 all I need from dvc queue status --json would be something like:

{
    "workers": { "active": number, "idle": number },
    ...
}

The rest of the output is inconsequential for us at the moment. As an interim solution I can proxy the required output by applying some regular expression to the currently provided string.

The only alternative solution that I can think of to this problem is adding the above information into exp show's workspace record. Although we have talked about not just bolting more and more data onto exp show in the past so I don't think this is the correct way to go.

For a future iteration:

This might be a good time to come back to the overall approach of the way the extension calls the CLI for experiment data. I remember that we discussed some kind of sub command that would show us all of the current experiments/checkpoints in the workspace. It would return this information without collecting any of the associated data. The extension could then use this information to only call exp show with the delta of what it currently has and what is required.

For this to work we would need to add some kind of dvc exp status command which would provide minimal data about the current experiments/checkpoints in the workspace. We would also have to add the ability to call exp show with ids as positional arguments. These ids could be any combination of shas/short shas/names. The reason that I bring this up is that dvc exp status would be a good place to put the number of workers.

WDYT?

[dberenbaum]

[Q] Will running dvc exp remove clean up the associated queue task(s) from dvc queue status?

It doesn't seem like this is the case from reading iterative/dvc.org#3715. If we are going to try and hide this concept then I think that is something that should be handled by either the CLI or extension. IMO it should be handled by default. If the user wants to keep the logs around then they can pass a flag.

🤔 Interesting question. dvc queue remove should not remove the associated exp from dvc exp show, but I don't think we have considered the reverse. I can see how you might still want to keep the queue task, but I agree that it seems like reasonable default behavior to clean it up when the experiment is removed. @karajan1001 Thoughts?

[karajan1001]

[Q] Will running dvc exp remove clean up the associated queue task(s) from dvc queue status?
It doesn't seem like this is the case from reading iterative/dvc.org#3715. If we are going to try and hide this concept then I think that is something that should be handled by either the CLI or extension. IMO it should be handled by default. If the user wants to keep the logs around then they can pass a flag.

🤔 Interesting question. dvc queue remove should not remove the associated exp from dvc exp show, but I don't think we have considered the reverse. I can see how you might still want to keep the queue task, but I agree that it seems like reasonable default behavior to clean it up when the experiment is removed. @karajan1001 Thoughts?

To simplify user experience, I think we should only keep exp and hide the whole queue sub-command. But had no time to go into deep. To achieve this, we need to solve the problem of one queue task creates hundreds of exp results.

[dberenbaum]

To simplify user experience, I think we should only keep exp and hide the whole queue sub-command. But had no time to go into deep. To achieve this, we need to solve the problem of one queue task creates hundreds of exp results.

Interesting thought. This sounds a little extreme, but maybe you meant this as a long-term goal and not a short-term one? I think it's a good goal to work towards, but let's expose the queue sub-command for now and then start to see how much of its functionality we can cover with dvc exp.

[dberenbaum]

Closing as duplicate of #1995

[karajan1001]

To simplify user experience, I think we should only keep exp and hide the whole queue sub-command. But had no time to go into deep. To achieve this, we need to solve the problem of one queue task creates hundreds of exp results.

Interesting thought. This sounds a little extreme, but maybe you meant this as a long-term goal and not a short-term one? I think it's a good goal to work towards, but let's expose the queue sub-command for now and then start to see how much of its functionality we can cover with dvc exp.

Yeah, of course we can't finish it soon, so let us continue on the current API.