-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Dataloader updates and streamlining #99
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
add 3b config, replace tele configs (for eventual back-porting of non-tele stuff)
This was referenced Jul 29, 2024
lchu6
approved these changes
Jul 30, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LG.
regression test passed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A number of cleanup/streamlining changes with no behavioral impact at the training level. This is mostly preparation for landing in torchdata and upcoming features, particularly support for n_workers > 1.
setup
fn, which executes immediately before saving/loading/stepping. This allows for modification of rank/path after instantiation.SamplingDataset
andScalableShardDataset
are now implemented as proper_WrapperDatasets
(though they still must stack in the same order and location). This reflects the intended modular use of_WrapperDatasets
, rather than treating these as edge cases of_StatefulDataset
due to their containing multiple sub-iterators. No more pass-through args!PreloadBufferDataset
so that when the buffer is too large (i.e. rescaling to a smaller number of workers) it will shrink down to the desired size over time, rather than staying oversized.Again, no user-facing behavior is changed, these are just preparatory updates for further additions. Backward compatibility with older checkpoints is maintained, and training code runs unaffected.