Training metrics become NaN after 2k iterations in stage1

[Asnly1]

Hello, thank you for your amazing work. I met a problem when trainning stage1 on my own dataset: all evaluation metrics become NaN after 2000 iterations. The training starts normally but fails to continue after this point. I have tried to prepare my data as provided data. Are there any specific data format requirements I should check? Or What may cause this problem?

[Asnly1]

I find that it is because new_mask0 consists of 0 without other value in human_loader.py's function get_rectified_stereo_data. Thus flow in the following part becomes 0 and results in NaN. Does anyone encounter this problem?

[pengcanon]

Did you rectify your training data offline? It seems as if you are doing rectification while training. If so, have you tried rectifying your training data first before conducting stage 1 training?