[feat,refactor,fix] Major change: multitasking, See details #173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ommit in description - This PR has breaking changes for API and will break a lot of things before v0.3.1 - Upgrades for 1.2.0 - Add support for MultiTasking, multiple datasets can be trained together now - Add proper version for fastText - Remove concept of tasks, datasets are first class citizens - Update the folder structure to reflect datasets as first class citizens - Fixes for Distributed setup
facebook-github-bot
added
the
CLA Signed
apsdehal
changed the title
apsdehal
changed the title
vedanuj
approved these changes
Aug 29, 2019
configs/vqa/vqa2/pythia.yml
Outdated
| vqa2: | ||
| image_features: | ||
| train: | ||
| - /private/home/asg/datasets/COCO/detectron_fix_100/fc6/train_val_2014,coco/resnet152/train_val_2014 |
| vqa2: | ||
| image_features: | ||
| train: | ||
| - /private/home/asg/datasets/COCO/detectron_fix_100/fc6/train_val_2014,coco/resnet152/train_val_2014 |
| self._load_imdb(imdb_path) | ||
|
|
||
| def _load_imdb(self, imdb_path): | ||
| if imdb_path.endswith(".npy"): | ||
| self._load_npy(imdb_path) | ||
| elif imdb_path.endswith(".jsonl"): | ||
| self._load_jsonl(imdb_path) | ||
| elif imdb_path.contains("visdial") or imdb_path.contains("visual_dialog"): |
There was a problem hiding this comment.
What is the difference between visdial and visual_dialog?
pythia/trainers/base_trainer.py
Outdated
| @@ -57,7 +58,7 @@ def _init_process_group(self): | |||
| raise RuntimeError( | |||
| "Unable to initialize process group: NCCL is not available" | |||
| ) | |||
| torch.distributed.init_process_group(backend="nccl") | |||
| torch.distributed.init_process_group(backend="nccl", init_method="env://") | |||
There was a problem hiding this comment.
init_method default value is env://. Do we need to specify explicitly?
pythia/datasets/multi_dataset.py
Outdated
| import numpy as np | ||
| from torch.utils.data import Dataset | ||
| from torch.utils.data import DataLoader | ||
| # from torch.utils.data.distributed import DistributedSampler |
apsdehal
added a commit
that referenced
this pull request
May 8, 2020
* [feat,refactor,bug] Major change: multitasking, See details for the commit in description - This PR has breaking changes for API and will break a lot of things before v0.3.1 - Upgrades for 1.2.0 - Add support for MultiTasking, multiple datasets can be trained together now - Add proper version for fastText - Remove concept of tasks, datasets are first class citizens - Update the folder structure to reflect datasets as first class citizens - Fixes for Distributed setup * [fix] Remove import for single dataset * [fix] Fix metrics name in configs * [fix] Fix values for tests due to changes in PyTorch 1.2 * [fix] Remove print statement in test * [fix] Address comments * [fix] Pythia train and val configuration * [fix] TextVQA config * Update multi_dataset.py * Update base_trainer.py
apsdehal
added a commit
that referenced
this pull request
May 8, 2020
* [feat,refactor,bug] Major change: multitasking, See details for the commit in description - This PR has breaking changes for API and will break a lot of things before v0.3.1 - Upgrades for 1.2.0 - Add support for MultiTasking, multiple datasets can be trained together now - Add proper version for fastText - Remove concept of tasks, datasets are first class citizens - Update the folder structure to reflect datasets as first class citizens - Fixes for Distributed setup * [fix] Remove import for single dataset * [fix] Fix metrics name in configs * [fix] Fix values for tests due to changes in PyTorch 1.2 * [fix] Remove print statement in test * [fix] Address comments * [fix] Pythia train and val configuration * [fix] TextVQA config * Update multi_dataset.py * Update base_trainer.py
facebook-github-bot
pushed a commit
that referenced
this pull request
May 8, 2021
Summary: Pull Request resolved: fairinternal/mmf-internal#173 * adds a feature in download.py to allow it to work for manifold * onboards clip_processor to work with all three config below: * local on disk file * manifold file * http file Reviewed By: vedanuj Differential Revision: D27760358 fbshipit-source-id: 1b7b8eff09a21e8afc48971d69d46df18a8ced6b
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
before v0.3.1
together now
citizens