Resumable model execution

[aristizabal95]

Hasasn suggested the possibility of caching predictions from model executions. This sparked a conversation on the topic of resumable model execution.

Feature:

Have predictions cached so that, if the model is interrupted mid-execution, it can be resumed next time.

Advantages:

• The execution can be stopped without losing progress
• Tools can be provided for executing results-only mlcubes, without having to go back and generate all predictions.

Disadvantages:

• Relies on model-owner implementation. Medperf is problem-agnostic, so we can't determine where a model should resume.
• Possibility of prediction collision and side-effects. Depends on how we implement things, but in general this added complexity could cause extra failure cases from the model.

[aristizabal95]

One idea to avoid the disadvantages would be to provide an extra input to the mlcube, which refers to previous predictions. If the model is executed for the first time, then this would be an empty folder. The next time it would contain the predictions obtained from the previous execution.

Advantages:

• Makes the cube idempotent.
• Avoids collision by separating the previous output from the new output.
• MLCubes that don't make use of this folder shouldn't present side effects

Disadvantages:

• Extra complexity for both medperf's code and model owner's logic. Given that MLCube is already a new idea, and one that could cause some friction, providing this could make it more of an obstacle to understand medperf's functionality.

[hasan7n]

Interesting!

Resumable at the MLCubes level, or MLCubes task-level :

I was first pointing to have a resumable execution at the MLCubes level, or MLCubes task-level:
We can make Medperf keep track of the last successful MLCube execution of a pipeline (preparation, model, metrics). So that, let's say a benchmark execution was interrupted during the execution of the metrics cube, only the metrics cube would be rerun. Predictions resulted from the model cube would have been cached.
Also, if we want to extend this to the task-level (makes sense only in the preparation cube), we can make Medperf keep track of the last successful task execution (prepare, check, statistics)...

But I like the idea of resumability at any point during the execution:

Resumable from any point:

(Correct me if I am wrong: this functionality expects that the interruption is caused by something outside of the mlcube itself. If the mlcube itself fails and interrupts the pipeline, I can only think of a corrupt mlcube that should be replaced.)

I like the idea of the extra input. We can investigate further to see what can be easy for mlcube authors. Quick thought: the extra input can be some integers referring to data indices. The cube also outputs the last data index it successfully processed..

Alternative

Do you think this can be helpful?
https://docs.docker.com/engine/reference/commandline/pause/
I wonder if this can be possible. This becomes an mlcube feature by the way. I am imagining that in case of (external) interruption of the mlcube execution, mlcube error handlers call docker pause. An argument to mlcube run can be added to specify if it should resume or restart..
But I am not fully aware of how this works and its side effects, e.g. are paused container still up running but frozen?

[aristizabal95]

I see! The good thing is that predictions are already stored in a uniquely identifiable way, so we can definitely start working on your initial thoughts. The one obstacle I see is determining wether the predictions were obtained after a successful execution, or if they're incomplete due to interrupted or failed mlcube execution.

As you suggest, we would need to keep track of execution status. This status could be on the level of (dataset, mlcube, task) tuple somewhere, and read it before the execution of any cube