From 994498ce7d66190fdcfffd0b4acc1a06afd2d7eb Mon Sep 17 00:00:00 2001 From: Kevin Tse Date: Fri, 24 Jun 2022 14:28:55 -0400 Subject: [PATCH] [Release 0.4.0] Update DataLoader2 README and fix broken links in doc (#554) * Small README update for DataLoader2 (#541) Summary: Pull Request resolved: https://github.com/pytorch/data/pull/541 Test Plan: Imported from OSS Reviewed By: ejguan Differential Revision: D37394421 Pulled By: NivekT fbshipit-source-id: 1fc9d47b44bc9348e3489f5e2a71a40010f527fc * Update changed links in documentation (#543) Summary: Pull Request resolved: https://github.com/pytorch/data/pull/543 Fixes #540 Test Plan: Imported from OSS Reviewed By: ejguan Differential Revision: D37394420 Pulled By: NivekT fbshipit-source-id: 6a29514885d64868956fa8c24cde958db1461b6b --- README.md | 10 +++++----- docs/source/examples.rst | 4 ++-- docs/source/tutorial.rst | 2 +- torchdata/dataloader2/README.md | 3 ++- 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index dbb2c10a3..7c122d649 100644 --- a/README.md +++ b/README.md @@ -204,14 +204,14 @@ A: Multi-process data loading is still handled by the `DataLoader`, see the [DataLoader documentation for more details](https://pytorch.org/docs/stable/data.html#single-and-multi-process-data-loading). As of PyTorch version >= 1.12.0 (TorchData version >= 0.4.0), data sharding is automatically done for DataPipes within the `DataLoader` as long as a `ShardingFiler` DataPipe exists in your pipeline. Please see the -[tutorial](https://pytorch.org/data/beta/tutorial.html#working-with-dataloader) for an example. +[tutorial](https://pytorch.org/data/main/tutorial.html#working-with-dataloader) for an example. Q: What is the upcoming plan for DataLoader? -A: There will be a new version of DataLoader in the next release. At the high level, the plan is that DataLoader V2 will -only be responsible for multiprocessing, distributed, and similar functionalities, not data processing logic. All data -processing features, such as the shuffling and batching, will be moved out of DataLoader to DataPipe. At the same time, -the current/old version of DataLoader should still be available and you can use DataPipes with that as well. +A: `DataLoader2` is in the prototype phase and more features are actively being developed. Please see the +[README file in `torchdata/dataloader2`](https://github.com/pytorch/data/blob/main/torchdata/dataloader2/README.md). If +you would like to experiment with it (or other prototype features), we encourage you to install the nightly version of +this library. Q: Why is there an Error saying the specified DLL could not be found at the time of importing `portalocker`? diff --git a/docs/source/examples.rst b/docs/source/examples.rst index 5385f4c0d..2493fab78 100644 --- a/docs/source/examples.rst +++ b/docs/source/examples.rst @@ -56,13 +56,13 @@ Vision Caltech 101 ^^^^^^^^^^^^^^^^^^^^^^^^^^ -The `Caltech 101 dataset `_ contains pictures of objects +The `Caltech 101 dataset `_ contains pictures of objects belonging to 101 categories. Here is the `DataPipe implementation of Caltech 101 `_. Caltech 256 ^^^^^^^^^^^^^^^^^^^^^^^^^^ -The `Caltech 256 dataset `_ contains 30607 images +The `Caltech 256 dataset `_ contains 30607 images from 256 categories. Here is the `DataPipe implementation of Caltech 256 `_. diff --git a/docs/source/tutorial.rst b/docs/source/tutorial.rst index e049aa0d2..41ac55e9e 100644 --- a/docs/source/tutorial.rst +++ b/docs/source/tutorial.rst @@ -171,7 +171,7 @@ Note: operations such as decoding, in order to avoid repeating these expensive operations across worker/distributed processes. -You can find more DataPipe implementation examples for various research domains `on this page `_. +You can find more DataPipe implementation examples for various research domains `on this page `_. Implementing a Custom DataPipe diff --git a/torchdata/dataloader2/README.md b/torchdata/dataloader2/README.md index 58fb96d80..9c11bb8b2 100644 --- a/torchdata/dataloader2/README.md +++ b/torchdata/dataloader2/README.md @@ -82,4 +82,5 @@ A sequence of graph utilities are provided to help users to define their own `Re ## Prototype Usage and Feedback `DataLoader2` is stable in terms of API, but functionally not complete yet. We welcome early adopters and feedback, as -well as potential contributors. +well as potential contributors. If you are interested in trying it out, we encourage you to install the nightly version +of this library.