Skip step if loss is nan

[chrisociepa]

There are various of reasons why loss might become nan. To prevent breaking training, we can just skip such step

[PershingSquare]

I would recommend against skipping training when the loss becomes NaN, as this usually indicates that there is some numerical instability in the model that needs to be addressed. Continuing to train the model in this state is unlikely to produce meaningful results and may even cause the model to perform worse.

Instead, it's important to investigate why the loss is becoming NaN and take appropriate steps to address the underlying issue. This could involve checking the data for errors or inconsistencies, adjusting the model architecture or hyperparameters, or using more stable numerical methods for training.

[karpathy]

agree... closing