Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download and Kinetics 400/600/700 Datasets #3680

Merged
merged 72 commits into from
Jun 10, 2021
Merged
Show file tree
Hide file tree
Changes from 68 commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
5e60e01
Initial commit
bjuncek Apr 16, 2021
62c5e05
pmeiers comments
bjuncek Apr 20, 2021
090e526
pmeiers changes
bjuncek Apr 20, 2021
9403bcf
pmeiers comments
bjuncek Apr 20, 2021
e08cf09
replace pandas with system library to avoid crashes
bjuncek Apr 22, 2021
29a4f03
Lint
bjuncek Apr 22, 2021
8cd5209
Lint
bjuncek Apr 22, 2021
a6d2490
fixing unittest
bjuncek Apr 22, 2021
c2076a7
Merge branch 'master' into bkorbar/datasets/kinetics
bjuncek Apr 22, 2021
93d1444
Minor comments removal
bjuncek Apr 22, 2021
9e3d3f3
pmeier comments
bjuncek Apr 30, 2021
139ec6d
remove asserts
bjuncek Apr 30, 2021
33a9f98
address pmeier formatting changes
bjuncek Apr 30, 2021
abdd2f6
address pmeier changes
bjuncek Apr 30, 2021
1460886
pmeier changes
bjuncek Apr 30, 2021
7b06906
rename n_classes to num_classes
bjuncek Apr 30, 2021
62f7fd5
Merge branch 'bkorbar/datasets/kinetics' of github.com:bjuncek/vision…
bjuncek Apr 30, 2021
e76f4ab
formatting changes
bjuncek Apr 30, 2021
0a8f216
doc change to add ".mp4" to backported class
bjuncek Apr 30, 2021
94a40aa
formatting to correct line length
bjuncek Apr 30, 2021
c585a5f
adding **kwargs to Kinetics400 class
bjuncek Apr 30, 2021
8cacd80
remove urlib request and download the file directly
bjuncek Apr 30, 2021
802f8f9
annotations and files can be already downloaded
bjuncek Apr 30, 2021
af70e5f
test fix
bjuncek May 4, 2021
6ec3253
add download tests for Kinetics
pmeier May 4, 2021
b84b298
users now dont need to provide full path within the root for new Kine…
bjuncek May 4, 2021
adbc2f8
Merge branch 'master' into bkorbar/datasets/kinetics
bjuncek May 4, 2021
d7f14d0
linter
bjuncek May 4, 2021
34c4323
Merge branch 'bkorbar/datasets/kinetics' of github.com:bjuncek/vision…
bjuncek May 4, 2021
96e2bec
Update test/test_datasets_download.py
pmeier May 5, 2021
20dc75d
Update torchvision/datasets/kinetics.py
bjuncek May 5, 2021
5ea1232
revert whitespace (3680#discussion_r626382842)
bjuncek May 5, 2021
607a3cb
addressing annotation_path parameter which is unnecessary
bjuncek May 5, 2021
da586c6
Update torchvision/datasets/kinetics.py
bjuncek May 5, 2021
23cc7f3
Merge branch 'bkorbar/datasets/kinetics' of github.com:bjuncek/vision…
bjuncek May 5, 2021
fd2208b
Update torchvision/datasets/kinetics.py
bjuncek May 5, 2021
0dc04d3
kwargs update
bjuncek May 5, 2021
d645f93
Merge branch 'bkorbar/datasets/kinetics' of github.com:bjuncek/vision…
bjuncek May 5, 2021
2bdd820
expose num_download_workers as public
bjuncek May 5, 2021
5640dd9
swap os.isfile with check_integrity
bjuncek May 5, 2021
9ef70da
nit on private things
bjuncek May 5, 2021
b7b81b1
special case if there are no default arguments
bjuncek May 5, 2021
36bd2c7
revert changes to kinetics400 test case for BC
bjuncek May 5, 2021
2bda79c
add split_folder changes and support for legacy format
bjuncek May 5, 2021
1a7a978
pmeiers suggestions
bjuncek May 11, 2021
89e41e6
pmeiers suggestions - root comment
bjuncek May 11, 2021
5941dab
pmeiers comments - annotation attribute remmoved
bjuncek May 11, 2021
72d260a
pmeiers suggestion
bjuncek May 11, 2021
51231cf
pmeiers suggestion
bjuncek May 11, 2021
7b91bbe
pmeiers suggestion
bjuncek May 11, 2021
cd2e55a
pmeiers suggestion
bjuncek May 11, 2021
7b322e9
Update torchvision/datasets/kinetics.py
bjuncek May 11, 2021
328c84e
Update torchvision/datasets/kinetics.py
bjuncek May 11, 2021
22e5d48
Update torchvision/datasets/kinetics.py
bjuncek May 11, 2021
173d385
Update torchvision/datasets/kinetics.py
bjuncek May 11, 2021
5a7db27
Update torchvision/datasets/kinetics.py
bjuncek May 11, 2021
44030ee
Update torchvision/datasets/kinetics.py
bjuncek May 11, 2021
ce5f80b
minor debugging
bjuncek May 11, 2021
803bab1
nit picks
bjuncek May 11, 2021
6e64bb6
only include public kwargs into defaults
pmeier May 12, 2021
8b64d1d
add _use_legacy_structure in favour of **kwargs
pmeier May 12, 2021
94b21cc
add type hints for Kinetics400
pmeier May 12, 2021
f803946
flake8
pmeier May 12, 2021
b39646a
flake8
pmeier May 12, 2021
c47c309
flake8
pmeier May 12, 2021
18ad36d
rename to make thigs clearer
bjuncek May 24, 2021
d0fa6f4
Merge branch 'master' into bkorbar/datasets/kinetics
pmeier May 25, 2021
61334f0
Merge branch 'master' into bkorbar/datasets/kinetics
pmeier Jun 7, 2021
12b76d7
permuting the output
bjuncek Jun 8, 2021
89b9bee
Merge branch 'master' into bkorbar/datasets/kinetics
bjuncek Jun 8, 2021
1ee00b2
Merge branch 'master' into bkorbar/datasets/kinetics
pmeier Jun 9, 2021
a22e4e7
Merge branch 'master' into bkorbar/datasets/kinetics
fmassa Jun 10, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions test/datasets_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -416,7 +416,11 @@ def _populate_private_class_attributes(cls):
continue

defaults.append(
{kwarg: default for kwarg, default in zip(argspec.args[-len(argspec.defaults):], argspec.defaults)}
{
kwarg: default
for kwarg, default in zip(argspec.args[-len(argspec.defaults):], argspec.defaults)
if not kwarg.startswith("_")
}
)

if not argspec.varkw:
Expand Down Expand Up @@ -637,7 +641,7 @@ def __init__(self, *args, **kwargs):

def _set_default_frames_per_clip(self, inject_fake_data):
argspec = inspect.getfullargspec(self.DATASET_CLASS.__init__)
args_without_default = argspec.args[1:-len(argspec.defaults)]
args_without_default = argspec.args[1:(-len(argspec.defaults) if argspec.defaults else None)]
frames_per_clip_last = args_without_default[-1] == "frames_per_clip"

@functools.wraps(inject_fake_data)
Expand Down
21 changes: 21 additions & 0 deletions test/test_datasets.py
Original file line number Diff line number Diff line change
Expand Up @@ -944,6 +944,27 @@ def test_not_found_or_corrupted(self):
super().test_not_found_or_corrupted()


class KineticsTestCase(datasets_utils.VideoDatasetTestCase):
DATASET_CLASS = datasets.Kinetics
ADDITIONAL_CONFIGS = datasets_utils.combinations_grid(
split=("train", "val"), num_classes=("400", "600", "700")
)

def inject_fake_data(self, tmpdir, config):
classes = ("Abseiling", "Zumba")
num_videos_per_class = 2
tmpdir = pathlib.Path(tmpdir) / config['split']
digits = string.ascii_letters + string.digits + "-_"
for cls in classes:
datasets_utils.create_video_folder(
tmpdir,
cls,
lambda _: f"{datasets_utils.create_random_string(11, digits)}.mp4",
num_videos_per_class,
)
return num_videos_per_class * len(classes)


class Kinetics400TestCase(datasets_utils.VideoDatasetTestCase):
DATASET_CLASS = datasets.Kinetics400

Expand Down
20 changes: 20 additions & 0 deletions test/test_datasets_download.py
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,25 @@ def widerface():
)


def kinetics():
return itertools.chain(
*[
collect_download_configs(
lambda: datasets.Kinetics(
path.join(ROOT, f"Kinetics{num_classes}"),
frames_per_clip=1,
num_classes=num_classes,
split=split,
download=True,
),
name=f"Kinetics, {num_classes}, {split}",
file="kinetics",
)
for num_classes, split in itertools.product(("400", "600", "700"), ("train", "val"))
]
)


def kitti():
return itertools.chain(
*[
Expand Down Expand Up @@ -440,6 +459,7 @@ def make_parametrize_kwargs(download_configs):
usps(),
celeba(),
widerface(),
kinetics(),
kitti(),
)
)
Expand Down
4 changes: 2 additions & 2 deletions torchvision/datasets/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from .sbd import SBDataset
from .vision import VisionDataset
from .usps import USPS
from .kinetics import Kinetics400
from .kinetics import Kinetics400, Kinetics
from .hmdb51 import HMDB51
from .ucf101 import UCF101
from .places365 import Places365
Expand All @@ -34,6 +34,6 @@
'Omniglot', 'SBU', 'Flickr8k', 'Flickr30k',
'VOCSegmentation', 'VOCDetection', 'Cityscapes', 'ImageNet',
'Caltech101', 'Caltech256', 'CelebA', 'WIDERFace', 'SBDataset',
'VisionDataset', 'USPS', 'Kinetics400', 'HMDB51', 'UCF101',
'VisionDataset', 'USPS', 'Kinetics400', "Kinetics", 'HMDB51', 'UCF101',
'Places365', 'Kitti',
)
Loading