Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytorch DataLoader mulit-process support #145

Closed
eddyxu opened this issue Sep 7, 2022 · 0 comments · Fixed by #147
Closed

Pytorch DataLoader mulit-process support #145

eddyxu opened this issue Sep 7, 2022 · 0 comments · Fixed by #147
Assignees
Labels
python PyTorch PyTorch support

Comments

@eddyxu
Copy link
Contributor

eddyxu commented Sep 7, 2022

Problem Statement

As per Pytorch document

https://pytorch.org/docs/stable/data.html#iterable-style-datasets

When num_workers > 0, each worker process will have a different copy of the dataset object, so it is often desired to configure each copy independently to avoid having duplicate data returned from the workers. get_worker_info(), when called in a worker process, returns information about the worker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python PyTorch PyTorch support
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant