Can I manually stop the trainning and find the best model

[jsantiagogonzalez]

Hi there!

I bring a simple question. I am just getting starting with nnUNet. I'm trainning the 1st fold (800th epoch). My question is: Will the model stop? Overfitting is clear at this point, but if I stop the trainning manually and go ahead with the other folds, the command nnUNet_find_best_model does not seem to work.

Any suggestion?
Thank you!

[jsantiagogonzalez]

I found the answer, you have to modify one the archives in nnUNet, so you have to compile code from source.

[jhogan7]

@jsantiagogonzalez what solution did you find? Also, it appears you can modify max_num_epochs online 48 of ~/nnUNet/nnunet/training/network_training/nnUNetTrainerV2.py.

[jsantiagogonzalez]

HI!

I just changed max_num_epoch from 1000 to 50, it works but obviously, as Fabian said, if overfitting is not very clear you should not do that

[FabianIsensee]

Just leave it. As long as the green line does not go down you do not have to worry about overfitting!
Do not change the number of epochs unless you have a very good reason to do so

[ahxmeds]

I started using nnUNet today and I have already put some jobs running for a custom 3D PET/CT dataset and now wondering if 1000 epochs is a little too much (since I am very close to my submission deadline).

@FabianIsensee Do you mean if I manually stop the run for any of the folds after say 500 epochs, will I still be able to access the best model from the mode_best.model file? The FAQ document says that after the training is over, the segmentation metrics will be written to summary.json. Will I be able to get generate this file later and get the segmentation metrics, if I manually stop the training run?

Please let me know and thanks in advance for your help.

[ynjiya]

@ahxmeds have you found the answer?
@FabianIsensee

[BennoGesierich]

@FabianIsensee you write that the checkpoint_best.pth could be used in case the green line goes down (you mean the pseudo dice in progress.png, right?). I was wondering, why is the best model not just always used as the default, instead of the latest model? Isn't the best model defined as the model with the peak on the green curve?

[FabianIsensee]

THe best model is determined on a proxy metric which is the pseudo dice. It is not perfectly reliable. Empirically the best model often does not perform better than the last. I have more trust in the last model which is why that's what is used by default

[BennoGesierich]

If one would perform a full validation after each epoch, do you think that this would change something? Would the "best model" then more likely perform better than the last model in an independent test sample?