Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DataPipe] Improving debug message when argument is a tuple/list of DataPipes #76134

Closed
wants to merge 5 commits into from

Conversation

NivekT
Copy link
Contributor

@NivekT NivekT commented Apr 20, 2022

Stack from ghstack:

The previous PR #75618 neglected to handle the case where the argument is a tuple/list of DataPipes, as it is the case for zip, mux, etc. This PR addresses that issue.

Now the message looks like the following, instead of a bunch of <...> inside the tuple.

This exception is thrown by __iter__ of ZipperIterDataPipe(datapipes=(IterableWrapperIterDataPipe, IterableWrapperIterDataPipe, IterableWrapperIterDataPipe))

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Apr 20, 2022

🔗 Helpful links

💊 CI failures summary and remediations

As of commit 0d221b4 (more details on the Dr. CI page):

Expand to see more
  • 1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

See GitHub Actions build pull / pytorch-xla-linux-bionic-py3.7-clang8 / test (xla, 1, 1, linux.2xlarge) (1/1)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

2022-04-22T16:21:36.1189384Z �[1A�[K�[31m�[1mER...clusion(s) in rule '@llvm-project//mlir:TableGen':
2022-04-22T16:21:35.9726023Z     [Scann] Compiling llvm/lib/Support/Twine.cpp
2022-04-22T16:21:36.1177300Z 
2022-04-22T16:21:36.1180445Z �[1A�[K
2022-04-22T16:21:36.1183169Z �[1A�[K
2022-04-22T16:21:36.1185860Z �[1A�[K
2022-04-22T16:21:36.1186681Z �[1A�[K
2022-04-22T16:21:36.1187061Z �[1A�[K
2022-04-22T16:21:36.1187411Z �[1A�[K
2022-04-22T16:21:36.1187791Z �[1A�[K
2022-04-22T16:21:36.1188136Z �[1A�[K
2022-04-22T16:21:36.1189384Z �[1A�[K�[31m�[1mERROR: �[0m/home/jenkins/.cache/bazel/_bazel_jenkins/b463291cb8b07b4bfde1e3a43733cd1a/external/llvm-project/mlir/BUILD.bazel:6250:11: Compiling mlir/lib/TableGen/Constraint.cpp failed: undeclared inclusion(s) in rule '@llvm-project//mlir:TableGen':
2022-04-22T16:21:36.1190808Z this rule is missing dependency declarations for the following files included by 'mlir/lib/TableGen/Constraint.cpp':
2022-04-22T16:21:36.1191549Z   'bazel-out/k8-opt-exec-50AE0418/bin/external/llvm-project/llvm/config.cppmap'
2022-04-22T16:21:36.1192226Z   'bazel-out/k8-opt-exec-50AE0418/bin/external/llvm-project/llvm/Demangle.cppmap'
2022-04-22T16:21:36.1192880Z   'bazel-out/k8-opt-exec-50AE0418/bin/external/llvm_terminfo/terminfo.cppmap'
2022-04-22T16:21:36.1193503Z   'bazel-out/k8-opt-exec-50AE0418/bin/external/llvm_zlib/zlib.cppmap'
2022-04-22T16:21:36.1194007Z �[32m[541 / 6,693]�[0m 12 actions, 10 running
2022-04-22T16:21:36.1194712Z     Compiling llvm/lib/TableGen/Record.cpp; 2s local, remote-cache
2022-04-22T16:21:36.1195315Z     Compiling llvm/lib/TableGen/TGParser.cpp; 0s local, remote-cache
2022-04-22T16:21:36.1195939Z     Compiling llvm/lib/TableGen/JSONBackend.cpp; 0s local, remote-cache
2022-04-22T16:21:36.1196601Z     @llvm-project//llvm:TableGen; 0s local, remote-cache

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

@NivekT NivekT added module: data torch.utils.data release notes: dataloader release notes category topic: improvements topic category labels Apr 20, 2022
@NivekT NivekT requested a review from ejguan April 20, 2022 20:25
…e/list of DataPipes"


The previous PR #75618 neglected to handle the case where the argument is a tuple/list of DataPipes, as it is the case for `zip`, `mux`, etc. This PR addresses that issue.

Now the message looks like the following, instead of a bunch of `<...>` inside the tuple.
```
This exception is thrown by __iter__ of ZipperIterDataPipe(datapipes=(IterableWrapperIterDataPipe, IterableWrapperIterDataPipe, IterableWrapperIterDataPipe))
```


[ghstack-poisoned]
NivekT added a commit that referenced this pull request Apr 20, 2022
…ataPipes

ghstack-source-id: a301a302da95879decf6febecb61771de903f048
Pull Request resolved: #76134
…e/list of DataPipes"


The previous PR #75618 neglected to handle the case where the argument is a tuple/list of DataPipes, as it is the case for `zip`, `mux`, etc. This PR addresses that issue.

Now the message looks like the following, instead of a bunch of `<...>` inside the tuple.
```
This exception is thrown by __iter__ of ZipperIterDataPipe(datapipes=(IterableWrapperIterDataPipe, IterableWrapperIterDataPipe, IterableWrapperIterDataPipe))
```


[ghstack-poisoned]
NivekT added a commit that referenced this pull request Apr 21, 2022
…ataPipes

ghstack-source-id: 3db71b38876f8b620b8c595cd7db8c5776bbdf3f
Pull Request resolved: #76134
…e/list of DataPipes"


The previous PR #75618 neglected to handle the case where the argument is a tuple/list of DataPipes, as it is the case for `zip`, `mux`, etc. This PR addresses that issue.

Now the message looks like the following, instead of a bunch of `<...>` inside the tuple.
```
This exception is thrown by __iter__ of ZipperIterDataPipe(datapipes=(IterableWrapperIterDataPipe, IterableWrapperIterDataPipe, IterableWrapperIterDataPipe))
```


[ghstack-poisoned]
NivekT added a commit that referenced this pull request Apr 21, 2022
…ataPipes

ghstack-source-id: d5573b85eb9bc8d8028d00f303ca5574292b0534
Pull Request resolved: #76134
Copy link
Contributor

@ejguan ejguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you

…e/list of DataPipes"


The previous PR #75618 neglected to handle the case where the argument is a tuple/list of DataPipes, as it is the case for `zip`, `mux`, etc. This PR addresses that issue.

Now the message looks like the following, instead of a bunch of `<...>` inside the tuple.
```
This exception is thrown by __iter__ of ZipperIterDataPipe(datapipes=(IterableWrapperIterDataPipe, IterableWrapperIterDataPipe, IterableWrapperIterDataPipe))
```


[ghstack-poisoned]
NivekT added a commit that referenced this pull request Apr 22, 2022
…ataPipes

ghstack-source-id: 684c16d50fa97651907bcae2c357fe44be8703c9
Pull Request resolved: #76134
@NivekT
Copy link
Contributor Author

NivekT commented Apr 22, 2022

@pytorchbot merge this please

@facebook-github-bot facebook-github-bot deleted the gh/nivekt/49/head branch April 26, 2022 14:17
facebook-github-bot pushed a commit that referenced this pull request Apr 26, 2022
…ataPipes (#76134)

Summary:
Pull Request resolved: #76134

Approved by: https://github.com/ejguan

Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/ed8e498c7058e0c9d61ffdf883daa3d10f9d2b43

Reviewed By: seemethere, osalpekar

Differential Revision: D35874484

Pulled By: NivekT

fbshipit-source-id: 1d983fc37a96277718d34c4a2154d033c6985db8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla signed module: data torch.utils.data release notes: dataloader release notes category topic: improvements topic category
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants