[BE Hackathon] Automatically generate datapipe.pyi in setup.py #290

NivekT · 2022-03-10T19:40:43Z

Stack from ghstack:

This should only land after pytorch/pytorch#73991

~~CI will fail until that PR is landed into Core's nightly.~~ This is working now.

Automatically run the generation of datapipe.pyi in setup.py and remove the interface (.pyi) file from version control

Differential Revision: D34939044

[ghstack-poisoned]

ghstack-source-id: e1be597c9e2e452652adbf3dbf6299585dea1c24 Pull Request resolved: #290

NivekT · 2022-03-10T22:31:25Z

Same question as the other PR:
~~Should we keep the .pyi file in Git VCS or not?~~

We will be removing the .pyi file from Git. Once the file is generated, it should persist even when a developer switches between Git branches. Regeneration is needed via python setup.py develop if new DataPipes are added.

…p.py" This should only land after pytorch/pytorch#73991 Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface file from version control [ghstack-poisoned]

ghstack-source-id: f13032f1c9e83ee31249a91dc99e8cfa2c3580a0 Pull Request resolved: #290

…p.py" This should only land after pytorch/pytorch#73991 CI will fail until that PR is landed into Core's nightly. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface file from version control [ghstack-poisoned]

ghstack-source-id: 9aee267a2d4227648053797726c6c62de3a68a2b Pull Request resolved: #290

…p.py" This should only land after pytorch/pytorch#73991 CI will fail until that PR is landed into Core's nightly. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control [ghstack-poisoned]

ghstack-source-id: 56c02953bec89c5eeb7bedc3bb73be7aced11c3e Pull Request resolved: #290

…p.py" This should only land after pytorch/pytorch#73991 CI will fail until that PR is landed into Core's nightly. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control [ghstack-poisoned]

ghstack-source-id: 1c8f1b84213fa7d3ec976a3f22e565b842489900 Pull Request resolved: #290

…p.py" This should only land after pytorch/pytorch#73991 CI will fail until that PR is landed into Core's nightly. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control [ghstack-poisoned]

NivekT · 2022-03-16T21:21:47Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

…p.py" This should only land after pytorch/pytorch#73991 ~~CI will fail until that PR is landed into Core's nightly.~~ This is working now. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044) [ghstack-poisoned]

NivekT · 2022-03-16T21:37:42Z

.github/workflows/domain_ci.yml

@@ -125,6 +125,9 @@ jobs:
      - name: Check out torchdata repository
        uses: actions/checkout@v2

+      - name: Install mssing dependency


Other domain libraries generally require and install requests (which is required by torchdata) but not torchaudio. Therefore we are installing it here prior to installing torchdata.

https://github.com/pytorch/audio/blob/main/setup.py

Note that I fixed the typo

…p.py" This should only land after pytorch/pytorch#73991 ~~CI will fail until that PR is landed into Core's nightly.~~ This is working now. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044) [ghstack-poisoned]

NivekT · 2022-03-16T21:40:57Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ejguan

LGTM with one comment about GHA workflow.

.github/workflows/domain_ci.yml

…p.py" This should only land after pytorch/pytorch#73991 ~~CI will fail until that PR is landed into Core's nightly.~~ This is working now. Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface (.pyi) file from version control Differential Revision: [D34939044](https://our.internmc.facebook.com/intern/diff/D34939044) [ghstack-poisoned]

NivekT · 2022-03-21T15:19:54Z

@NivekT has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

pmeier · 2022-03-22T08:08:08Z

This breaks torchvision CI. Since there is no nightly release that I'm aware of, torchvision installs torchdata with pip install git+https://github.com/pytorch/data.

This PR is problematic, because you now have a build time dependency on requirements.txt as well as torch that is not picked up by pip. Thus, we need to now install with

pip install -r https://raw.githubusercontent.com/pytorch/data/main/requirements.txt
pip install --no-build-isolation git+https://github.com/pytorch/data

ejguan · 2022-03-22T13:17:36Z

Since there is no nightly release that I'm aware of, torchvision installs torchdata with pip install git+https://github.com/pytorch/data.

I am adding a nightly release by the end of this week. Does it help if nightly release is available?

pmeier · 2022-03-22T13:45:09Z

Does it help if nightly release is available?

Yes. Installing from a git repo means that we are installing from source. Thus, we need to have all build requirements available without pip knowing what they are. If you have a nightly release, we can simply install the binary and pip automatically picks up all runtime requirements.

ejguan · 2022-03-22T13:46:26Z

Does it help if nightly release is available?

Yes. Installing from a git repo means that we are installing from source. Thus, we need to have all build requirements available without pip knowing what they are. If you have a nightly release, we can simply install the binary and pip automatically picks up all runtime requirements.

Sure. Then, I will try to land nightly workflow asap

NivekT · 2022-03-22T14:15:07Z

Does it help if nightly release is available?

Yes. Installing from a git repo means that we are installing from source. Thus, we need to have all build requirements available without pip knowing what they are. If you have a nightly release, we can simply install the binary and pip automatically picks up all runtime requirements.

I didn't realize that was the case. Thanks for spotting that and fixing it.

ejguan · 2022-03-22T14:18:22Z

Does it help if nightly release is available?

Yes. Installing from a git repo means that we are installing from source. Thus, we need to have all build requirements available without pip knowing what they are. If you have a nightly release, we can simply install the binary and pip automatically picks up all runtime requirements.

I didn't realize that was the case. Thanks for spotting that and fixing it.

The doc generation workflow is failing because of the same reason. https://github.com/pytorch/data/runs/5633692990?check_suite_focus=true
cc: @NivekT

Fixing the issue [mentioned here](#290 (comment)). I have ran it on my fork and the build worked: https://github.com/NivekT/data/actions/runs/2023027716 Differential Revision: [D35048050](https://our.internmc.facebook.com/intern/diff/D35048050) [ghstack-poisoned]

Summary: Pull Request resolved: #314 Fixing the issue [mentioned here](#290 (comment)). I have ran it on my fork and the build worked: https://github.com/NivekT/data/actions/runs/2023027716 Test Plan: Imported from OSS Reviewed By: ejguan Differential Revision: D35048050 Pulled By: NivekT fbshipit-source-id: dcacd514c0138519e4581539fb4b019989d96a2a

Fixing the issue [mentioned here](#290 (comment)). I have ran it on my fork and the build worked: https://github.com/NivekT/data/actions/runs/2023027716 Differential Revision: [D35048050](https://our.internmc.facebook.com/intern/diff/D35048050) [ghstack-poisoned]

Automatically generate datapipe.pyi in setup.py

0bdf1e2

[ghstack-poisoned]

NivekT added a commit that referenced this pull request Mar 10, 2022

Automatically generate datapipe.pyi in setup.py

b3874a7

ghstack-source-id: e1be597c9e2e452652adbf3dbf6299585dea1c24 Pull Request resolved: #290

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 10, 2022

NivekT changed the title ~~Automatically generate datapipe.pyi in setup.py~~ [BE Hackathon] Automatically generate datapipe.pyi in setup.py Mar 10, 2022

NivekT marked this pull request as draft March 10, 2022 19:42

NivekT requested review from ejguan and VitalyFedyunin March 10, 2022 22:17

Update on "[BE Hackathon] Automatically generate datapipe.pyi in setu…

8ed0d92

…p.py" This should only land after pytorch/pytorch#73991 Automatically run the generation of `datapipe.pyi` in `setup.py` and remove the interface file from version control [ghstack-poisoned]

NivekT added a commit that referenced this pull request Mar 10, 2022

Automatically generate datapipe.pyi in setup.py

2178d63

ghstack-source-id: f13032f1c9e83ee31249a91dc99e8cfa2c3580a0 Pull Request resolved: #290

NivekT marked this pull request as ready for review March 10, 2022 22:35

NivekT added a commit that referenced this pull request Mar 11, 2022

Automatically generate datapipe.pyi in setup.py

fb56cf4

ghstack-source-id: 9aee267a2d4227648053797726c6c62de3a68a2b Pull Request resolved: #290

NivekT added a commit that referenced this pull request Mar 16, 2022

Automatically generate datapipe.pyi in setup.py

e28b74a

ghstack-source-id: 56c02953bec89c5eeb7bedc3bb73be7aced11c3e Pull Request resolved: #290

NivekT added a commit that referenced this pull request Mar 16, 2022

Automatically generate datapipe.pyi in setup.py

0cc245a

ghstack-source-id: 1c8f1b84213fa7d3ec976a3f22e565b842489900 Pull Request resolved: #290

NivekT mentioned this pull request Mar 16, 2022

Rename BucketBatcher argument to avoid name collision #304

Closed

NivekT commented Mar 16, 2022

View reviewed changes

ejguan approved these changes Mar 21, 2022

View reviewed changes

.github/workflows/domain_ci.yml Outdated Show resolved Hide resolved

facebook-github-bot closed this in 55ac7e3 Mar 21, 2022

pmeier mentioned this pull request Mar 22, 2022

fix torchdata CI installation pytorch/vision#5657

Merged

NivekT mentioned this pull request Mar 22, 2022

Fixing doc build workflow #314

Closed

facebook-github-bot deleted the gh/NivekT/62/head branch March 25, 2022 14:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BE Hackathon] Automatically generate datapipe.pyi in setup.py #290

[BE Hackathon] Automatically generate datapipe.pyi in setup.py #290

NivekT commented Mar 10, 2022 •

edited

Loading

NivekT commented Mar 10, 2022 •

edited

Loading

NivekT commented Mar 16, 2022

NivekT Mar 16, 2022

NivekT Mar 17, 2022

NivekT commented Mar 16, 2022

ejguan left a comment

NivekT commented Mar 21, 2022

pmeier commented Mar 22, 2022

ejguan commented Mar 22, 2022

pmeier commented Mar 22, 2022

ejguan commented Mar 22, 2022

NivekT commented Mar 22, 2022

ejguan commented Mar 22, 2022

[BE Hackathon] Automatically generate datapipe.pyi in setup.py #290

[BE Hackathon] Automatically generate datapipe.pyi in setup.py #290

Conversation

NivekT commented Mar 10, 2022 • edited Loading

NivekT commented Mar 10, 2022 • edited Loading

NivekT commented Mar 16, 2022

NivekT Mar 16, 2022

Choose a reason for hiding this comment

NivekT Mar 17, 2022

Choose a reason for hiding this comment

NivekT commented Mar 16, 2022

ejguan left a comment

Choose a reason for hiding this comment

NivekT commented Mar 21, 2022

pmeier commented Mar 22, 2022

ejguan commented Mar 22, 2022

pmeier commented Mar 22, 2022

ejguan commented Mar 22, 2022

NivekT commented Mar 22, 2022

ejguan commented Mar 22, 2022

NivekT commented Mar 10, 2022 •

edited

Loading

NivekT commented Mar 10, 2022 •

edited

Loading