model.eval() seems not work well

[mapleneverfade]

It's very kind of your work ! There's still something i want for your help!
My code for model and criterion parallel like this:
model = encoding.parallel.ModelDataParallel(model,device_ids=[0,1,2])
criterion = encoding.parallel.CriterionDataParallel(criterion,device_ids=[0,1,2])
Training process is going well, but when i turn to model.eval(), i got explosion of loss.

Is there something wrong with model.eval()?

[zhanghang1989]

Are you training with evaluation mode?

[mapleneverfade]

I set model.train() before every epoch of loader.

[zhanghang1989]

Thx for clarifying! What is the mIoU you were expecting? For example, what is the mIoU when using standard BatchNorm at 1st epoch?

[mapleneverfade]

[zhanghang1989]

Am I missing something. You said, "i turn to model.eval(), i got explosion of loss."
What does it mean? We typically do not calculate loss in evaluation mode.

[mapleneverfade]

Maybe I didn't clear it out. When I use standard BatchNorm I get normal loss both in train mode and eval mode.
While using syn-bn I get norm loss during training but explode in eval mode, it means there maybe
something wrong with syn-bn when turned it to eval mode.
I test the syn-bn model in model.eval(), got this

Calculating eval loss is not what i expect to, I just want to figure out what causes the test IoU to be zero.
Or is there something I misused？

[zhanghang1989]

Please checkout the PyTorch compatible Synchronized Cross-GPU encoding.nn.BatchNorm2d and the example.

[zhanghang1989]

@mapleneverfade The sycBN works the same as standard BN in eval mode. Please try the new PyTorch DataParallel compatible version.

[mapleneverfade]

I really appreciate for your replying.
But I still don't figure out what's going wrong.

Combination of "ModelDataParallel() & CriterionDataParallel() & Standard BN" goes well,
when I replace the "Standard BN" with "encoding.nn.BatchNorm2d",eval-mode still don't work.

[zhanghang1989]

Hi, have you checked out the example https://github.com/zhanghang1989/PyTorch-SyncBatchNorm

[zhanghang1989]

Hi @mapleneverfade , do you still have the problem? I still couldn't get it why do you calculate loss in eval mode