Skip to content

Commit

Permalink
[Release 0.4.0] Update DataLoader2 README and fix broken links in doc (
Browse files Browse the repository at this point in the history
…#554)

* Small README update for DataLoader2 (#541)

Summary: Pull Request resolved: #541

Test Plan: Imported from OSS

Reviewed By: ejguan

Differential Revision: D37394421

Pulled By: NivekT

fbshipit-source-id: 1fc9d47b44bc9348e3489f5e2a71a40010f527fc

* Update changed links in documentation (#543)

Summary:
Pull Request resolved: #543

Fixes #540

Test Plan: Imported from OSS

Reviewed By: ejguan

Differential Revision: D37394420

Pulled By: NivekT

fbshipit-source-id: 6a29514885d64868956fa8c24cde958db1461b6b
  • Loading branch information
NivekT authored Jun 24, 2022
1 parent 6cf5d05 commit 994498c
Show file tree
Hide file tree
Showing 4 changed files with 10 additions and 9 deletions.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,14 +204,14 @@ A: Multi-process data loading is still handled by the `DataLoader`, see the
[DataLoader documentation for more details](https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading).
As of PyTorch version >= 1.12.0 (TorchData version >= 0.4.0), data sharding is automatically done for DataPipes within
the `DataLoader` as long as a `ShardingFiler` DataPipe exists in your pipeline. Please see the
[tutorial](https://pytorch.org/data/beta/tutorial.html#working-with-dataloader) for an example.
[tutorial](https://pytorch.org/data/main/tutorial.html#working-with-dataloader) for an example.

Q: What is the upcoming plan for DataLoader?

A: There will be a new version of DataLoader in the next release. At the high level, the plan is that DataLoader V2 will
only be responsible for multiprocessing, distributed, and similar functionalities, not data processing logic. All data
processing features, such as the shuffling and batching, will be moved out of DataLoader to DataPipe. At the same time,
the current/old version of DataLoader should still be available and you can use DataPipes with that as well.
A: `DataLoader2` is in the prototype phase and more features are actively being developed. Please see the
[README file in `torchdata/dataloader2`](https://github.com/pytorch/data/blob/main/torchdata/dataloader2/README.md). If
you would like to experiment with it (or other prototype features), we encourage you to install the nightly version of
this library.

Q: Why is there an Error saying the specified DLL could not be found at the time of importing `portalocker`?

Expand Down
4 changes: 2 additions & 2 deletions docs/source/examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -56,13 +56,13 @@ Vision

Caltech 101
^^^^^^^^^^^^^^^^^^^^^^^^^^
The `Caltech 101 dataset <http://www.vision.caltech.edu/Image_Datasets/Caltech101/>`_ contains pictures of objects
The `Caltech 101 dataset <https://data.caltech.edu/records/20086>`_ contains pictures of objects
belonging to 101 categories. Here is the
`DataPipe implementation of Caltech 101 <https://github.com/pytorch/data/blob/main/examples/vision/caltech101.py>`_.

Caltech 256
^^^^^^^^^^^^^^^^^^^^^^^^^^
The `Caltech 256 dataset <http://www.vision.caltech.edu/Image_Datasets/Caltech256/>`_ contains 30607 images
The `Caltech 256 dataset <https://data.caltech.edu/records/20087>`_ contains 30607 images
from 256 categories. Here is the
`DataPipe implementation of Caltech 256 <https://github.com/pytorch/data/blob/main/examples/vision/caltech256.py>`_.

Expand Down
2 changes: 1 addition & 1 deletion docs/source/tutorial.rst
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ Note:
operations such as decoding, in order to avoid repeating these expensive operations across worker/distributed processes.


You can find more DataPipe implementation examples for various research domains `on this page <torchexamples.html>`_.
You can find more DataPipe implementation examples for various research domains `on this page <examples.html>`_.


Implementing a Custom DataPipe
Expand Down
3 changes: 2 additions & 1 deletion torchdata/dataloader2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,4 +82,5 @@ A sequence of graph utilities are provided to help users to define their own `Re
## Prototype Usage and Feedback

`DataLoader2` is stable in terms of API, but functionally not complete yet. We welcome early adopters and feedback, as
well as potential contributors.
well as potential contributors. If you are interested in trying it out, we encourage you to install the nightly version
of this library.

0 comments on commit 994498c

Please sign in to comment.