Learning rate scheduler should step after each optimizer step

[rmrao]

🐛 Bug

I'm not sure that this is a bug or if it is a deliberate design decision, but right now the learning rate schedule gets updated at every "step" which actually corresponds to every forward pass. I think a more standard implementation would have the learning rate scheduler "step" interval correspond to being updated every backwards pass. This has caused me a lot of problems with instability as I did not realize that using standard learning rate warmups of say 16000 steps would actually only warm up for 1000 steps if I set accumulate_grad_batches=16.

[williamFalcon]

good point. it should be every backward pass as you mention.

[williamFalcon]

mind submitting PR?

[rmrao]

Sure, will do.