g_loss is NaN cause of model.predictor_encoder and model.decoder

[xorium]

Hi! Thank you for you work!

I'm trying to run second stage training (single speaker, non-english language) and already in epoch 0, the code hits the line set_trace(). I checked the recommendations from here, and it seems that none of the points apply (I’m definitely using multilingual-PL-BERT, it’s epoch 0, I haven’t changed the code, and the training in the first stage went without any errors).

So I tried debugging the code to see the reason, and I noticed that everything starts with model.predictor_encoder and model.decoder returning NaN tensors with non-NaN input data.

I don't have enough knowledge to understand the cause further.
Could you at least guide me on where I should look next?
Thank you!

[martinambrus]

try checking this issue along with its pull request to see if it helps - #254

[xorium]

Thank you! This really helps!