-
Notifications
You must be signed in to change notification settings - Fork 23.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DataLoader] Locking lower ranks seed recepients #81071
[DataLoader] Locking lower ranks seed recepients #81071
Conversation
[ghstack-poisoned]
🔗 Helpful links
❌ 1 New FailuresAs of commit 8761195 (more details on the Dr. CI page): Expand to see more
🕵️ 1 new failure recognized by patternsThe following CI failures do not appear to be due to upstream breakages
|
[ghstack-poisoned]
ghstack-source-id: 32ab45be78eb2695600c5b46a7a3286340ba5bc8 Pull Request resolved: #81071
@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! LGTM
Exit seed receiving section only when all ranks received seed, otherwise we are at risk that current rank will reach same section of the code again while rank zero still in the previous iteration Fixes: #80845 Differential Revision: [D37702557](https://our.internmc.facebook.com/intern/diff/D37702557) [ghstack-poisoned]
ghstack-source-id: aceebd9b63eb9875f7d8f2313372fb7a4e9dd9d1 Pull Request resolved: #81071
@pytorchbot merge -g |
@pytorchbot successfully started a merge job. Check the current status here |
Merge failed due to This PR has internal changes and must be landed via Phabricator |
@VitalyFedyunin has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
@pytorchbot merge -g |
@pytorchbot successfully started a merge job. Check the current status here |
Hey @VitalyFedyunin. |
Summary: Exit seed receiving section only when all ranks received seed, otherwise we are at risk that current rank will reach same section of the code again while rank zero still in the previous iteration Fixes: #80845 Pull Request resolved: #81071 Approved by: https://github.com/msaroufim, https://github.com/ejguan Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/e9b3bc2eadb8ffe10c002abcd5a34a5b7d36f390 Original Phabricator Test Plan: Imported from OSS Reviewed By: mehtanirav, ejguan Differential Revision: D37702557 Pulled By: VitalyFedyunin fbshipit-source-id: 51dd950e1bfc2c984a4ddbe6481e225023b0a202
…orch#81071) Summary: Exit seed receiving section only when all ranks received seed, otherwise we are at risk that current rank will reach same section of the code again while rank zero still in the previous iteration Fixes: pytorch#80845 Pull Request resolved: pytorch#81071 Approved by: https://github.com/msaroufim, https://github.com/ejguan Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/e9b3bc2eadb8ffe10c002abcd5a34a5b7d36f390 Original Phabricator Test Plan: Imported from OSS Reviewed By: mehtanirav, ejguan Differential Revision: D37702557 Pulled By: VitalyFedyunin fbshipit-source-id: 51dd950e1bfc2c984a4ddbe6481e225023b0a202
Exit seed receiving section only when all ranks received seed, otherwise we are at risk that current rank
will reach same section of the code again while rank zero still in the previous iteration
Stack from ghstack (oldest at bottom):
Fixes: #80845
Differential Revision: D37702557