Sinusoidal Train/Val Accuracy

[leekezar]

I'm noticing that the transformer and the SL-GCN architectures, while learning on WLASL2000, have an accuracy curve that resembles a sine curve with period of about 20 epochs and amplitude of about 5-10%. I am using the example config provided in the repo, and verified that the batches are being shuffled. I have also played around with logging on_step=True in case this is an artifact of torch.nn.log, but that didn't help either. Any ideas why this is happening?

[GokulNC]

Hello @leekezar !

This is due to the CosineAnnealing LR Scheduler:

OpenHands/examples/configs/wlasl/decoupled_gcn.yaml

Lines 109 to 113 in 7fb2c0a

	scheduler:
	name: CosineAnnealingLR
	params:
	last_epoch: -1
	T_max: 10

The oscillation usually becomes negligible after a good no. of epochs.
But you can also remove this from the config. Shouldn't affect the results too much I believe.