Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support scaled optimizer state in distributed Adam optimizer #1771

Merged
merged 15 commits into from
Feb 8, 2024
Merged

Support scaled optimizer state in distributed Adam optimizer #1771

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d128d31
Add distopt support for scaled states
timmoon10 Jan 13, 2024
f1f36ba
Debug distopt checkpointing with scaled optimizer state
timmoon10 Jan 16, 2024
39d9e23
Debug inconsistent variable name
timmoon10 Jan 17, 2024
f9c7898
Debug checkpointing
timmoon10 Jan 17, 2024
67fd40f
Complain if scaling fp32 states
timmoon10 Jan 17, 2024
dc8d2cc
Make sure state scaling is done in fp32
timmoon10 Jan 17, 2024
3f9f724
Change from per-parameter scaling factors to per-fragment
timmoon10 Jan 18, 2024
6baf21e
Support overlapping first bucket AG with scaled state
timmoon10 Jan 19, 2024
27f261a
Correctly load in per-param-group settings from checkpoint
timmoon10 Jan 19, 2024
4d19c3b
Handle with contiguous param buffer and int param sync dtype
timmoon10 Jan 19, 2024
98b9fa3
Tweak docs
timmoon10 Jan 19, 2024
11d0211
Fix excessive memory usage with scaled optim state
timmoon10 Jan 19, 2024
fe7b812
Silence warning about autograd through broadcast
timmoon10 Jan 31, 2024
847296e
Debug tests with multiple models
timmoon10 Feb 6, 2024
4b96528
Merge branch 'master' into distopt-scaled-state
timmoon10 Feb 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading