weight decay for the resweight?

[Kyeongpil]

Hello, I read the paper, and it is interesting to me.
I have a question.

Many implements including Huggingface exclude LayerNorm and biases when decaying weights for convergence.
(huggingface/transformers#492)
Is it helpful to exclude the resweight parameters when decaying weights??

[calclavia]

Yes, it would seem reasonable to not decay resweights since other parameters are already being decayed.

[fightnyy]

@calclavia I have the same question, but did this prove to be better? Or is it just to speed up calculations?