The result of swin-small backbone on ADE

[zhihou7]

Hi,

I run Mask2Former on ADE (maskformer2_swin_small_bs16_160k.yaml) with 4 16GB V-100 GPUs. However, I can only achieve 49.6%, which is much worse than the reported result (51.3%). Could you provide the log for me to analysize the result?

Thanks

[bowenc0221]

ADE20K (and Cityscapes) dataset has very large variance, please try to train the model for 5 times, the median results should match the one we reported.

[zhihou7]

Thanks for your reply, I will try to train the network again. Is it because I just use 4 GPUs while you train the network with 8 GPUs?

[bowenc0221]

As long as you used the same batch size, the number of GPU does not matter.

[zhihou7]

Thanks for your reply.

I have trained the network multiple times and the best result is 50.9. I find the variance is significantly large.

[zhihou7]

Hi, @bowenc0221 , would you mind providing the variance for those methods and dataset? It is difficult to evaluate the method to compare the methods since the variance is so large.

[bowenc0221]

I don't have the variance for Mask2Former, but for the ADE20K and Cityscapes dataset we report the median of 3 runs. You can find the standard deviation (std) on the ADE20K dataset in our MaskFormer paper (https://arxiv.org/abs/2107.06278). Mask2Former should have a similar std.

[zhihou7]

Thanks for your reply.