Dice scores on test data

[ironb25]

Continuing my issue #65 (comment) here .
I checked the modalities order, its same as BRATS. also the labeling is in same order after visual check. its quite weird.. Any other suggestions maybe? I will explore more on this on what could be the issue.

On another note, I tried transfer learning (even though both BRATS and my dataset are HGG) as another sanity check and was getting dice scores around ~0.8 for all 3 classes. But doing just inference results in poor dice scores.

[FabianIsensee]

can you please provide one of your test set images? I cannot help you without seeing the data. You said you would expect a BraTS network to perform well on them? I have some of those around here and can try them out

[ironb25]

I found out the issue finally after spending some time. The json file in my dataset was based on this BRATS data conversion script https://github.com/MIC-DKFZ/nnUNet/blob/master/nnunet/dataset_conversion/Task032_BraTS_2018.py

json_dict['modality'] = {
        "0": "T1",
        "1": "T1ce",
        "2": "T2",
        "3": "FLAIR"
    }

So I was using this order of modalities for prediction, but infact the order of modalitites when you download dataset is different as I found out through the dataset.json file in Task0001_braintumour.
I changed the order of my dataset to this one, and got good prediction dice scores.

Your earlier suggestion of the issue with order oof modalitites was correct :)

[FabianIsensee]

glad to hear it works now ;-)

[JulianKlug]

I have another issue that would go under this title.
Somehow, I get good scores on test data (mean dice 0.78), whereas the nnunet seems not to train correctly when I look at the progress and cross-validation metrics (mean dice 0.25).

Is this just pure luck by test set selection or is something else going on?
I have my test set in the /imagesTs subfolder, I hope these are not used for the training or selection process?

[FabianIsensee]

There is nothing I can do from here to analyze the problem. It has to do with your data. You should to look at your data to answer this question. Things you can consider:

• are test cases also contained in the training set?
• are my annotations good? It looks like your segmentation problem is either very difficult (too little training data) or not consistently annotated
• is there a domsin shift between training and test set?
  Best,
  Fabian

[JulianKlug]

Thank you for your prompt answer! I am indeed working on a very difficult segmentation problem on a quite heterogenous population (ischemic stroke prediction). However there should be neither overlap nor domain shift between training and test set, but I will investigate this further to be sure!

[FabianIsensee]

Maybe also verify that nnU-Net is reading the images properly. Open up the preprocessed npz/npy files and double check that all annotations are in the correct place. Since it seems to be learning something it could be that only some few images are affected

[FabianIsensee]

Have a look (visual inspection)at the cross-validation predictions as well

[Peaceandmaths]

I have an inverse problem : during training the pseudo-dice is reaching about 85%, but when I test the model on test set I get only 45% dice score. What could be the cause for this discrepancy ? Any thing I can do to figure out the issue ? Any advice would help 🙏

I am training nnunet on 1400 CT images of the brain and segmenting aneurysms (small vessel malformations).