Can DeepSpeed ZeRO-3 be applied for training?

[avionkmh]

🌟 New model addition

We have applied DeepSpeed v0.3.10(ZeRO-2) on T5 training.
I heard DeepSpeed ZeRO-3 library has been released 10 days ago(8/MAR).
I'd like to adopt ZeRO-3 for our training.
Can this library be applied, especially for T5 training?
Do you have any experience applying this library?
If any, could share your experience?

[LysandreJik]

Hi! You might find this reply #10789 (comment) by @stas00 of interest.

[stas00]

@avionkmh, very soon it'll be supported, you may want to track: #10753

[avionkmh]

@stas00, @LysandreJik, thank you for your kind information. I'm going to track the issues you recommended.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.