Sparse to Dense training

[naston]

I am super interested in the sparse to dense training method proposed in the Monarch Matrices paper and had some questions about implementation, specifically surrounding the optimizer and scheduler. Did you have to re-initialize the optimizer to work with the new dense weights or were you able to convert the the optimizer states from sparse to dense as well? Similarly, did you find that you needed warmup when starting the dense phase of training or were you simply able to resume the scheduler with no issue? Lastly, I see that there have been a few other structured matrices introduced within this repository, in terms of training throughput should I use one of these matrix types instead of monarch?

Links to/mentions of relevant .py files within this matrix would also be appreciated.