Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Refactoring #59

Merged
merged 61 commits into from
Dec 14, 2023
Merged

WIP: Refactoring #59

merged 61 commits into from
Dec 14, 2023

Conversation

BerndDoser
Copy link
Member

@BerndDoser BerndDoser commented Dec 7, 2023 •
edited
Loading

Comprehensive refactoring of the data modules and the HiPSter in preparation for the applicability of different data structures.

Derived class DatasetWithMetadata to provide metadata

The dataset classes return the pure data tensor for training. The derived classes DatasetWithMetadata return a tuple of data and metadata, which is currently only needed for the HiPSter. This ensures that the access also works for shuffled dataloaders.

Move data structure-dependent functionality from HiPster into DataModule

Move write_catalog, create_images and create_thumbnails from HiPster into DataModule

Rotation invariant

The best rotation search function find_best_rotation is moved to the base class SpherinatorModule to avoid duplicated code for training and HiPSter. (here)

Bugfix: No variational sampling in rotational invariance

The variational sampling should be disabled in searching for the best rotation (encode instead of forward).

Base classes SpherinatorDataset and SpherinatorDataModule

Abstract base classes for dataset and DataModules to ensure that all methods for HiPSter are implemented.

Miscellaneous

@BerndDoser
black formatting on save
8f723ed
@BerndDoser
SpherinatorDataset: separate image and metadata
9c2ea83
@BerndDoser
test hipster generate hips
878d031
@BerndDoser
add missing import hipster
96f3236
@BerndDoser
use temporary directory for pytest
1ec93b6
@BerndDoser
Merge branch 'HITS-AIN:main' into data
93274e3
@BerndDoser
return indices from dataset.__getitem__
518221c 
for metadata
@BerndDoser
poetry: include hipster script
07ee7e8
@BerndDoser
more fits for testing IllustrisSdssDataModule
0c5d545
@BerndDoser
Merge branch 'HITS-AIN:main' into data
f789625
@BerndDoser
clean-up illustris dataset and -module
724e2e9 
and preprocessing

- no direct tensor support in dataset
- preprocessing: replace directory by images
- transforms.ToTensor
@BerndDoser
Merge branch 'data' of https://github.com/BerndDoser/Spherinator into…
f90568d 
… data
@BerndDoser
fix pytest errors and warnings
3666b28 
sort listdir to reproduce dataset filenames
@BerndDoser
add jsonargparse extra signatures
4aabe2f
@BerndDoser
Set antialias to True
223fe70 
pytorch/vision#7093
@BerndDoser
use clip instead of min/max transformation
306017e
@BerndDoser
clean-up experiments
0a831b9
@BerndDoser
Change transforms.ToTensor() into torch.Tensor()
012b9cb
@BerndDoser
test torch.BatchSampler
a36b9ca
@BerndDoser
not returning index from dataset getitem
65bea45 
+ use torchvision.transforms.v2
+ add dataloaders processing, images, thumbnail_images
+ test jit.script
@BerndDoser
Base class SpherinatorDataModule and
423f4ed 
GalaxyZooDataModule
@BerndDoser
store current index in dataset for metadata access
64862c4
@BerndDoser
hipster list of execution tasks
92da5f9
@BerndDoser
move images to docs
f2f5163
@BerndDoser
fix image location in README.md
7dc4846
@BerndDoser
use caret requirements for poetry dependencies
9db4790
@BerndDoser
torch load map location
66c5125
@BerndDoser
add create_images and create_allsky to hipster class
f7a997c 
- split member function using Mixin
(https://www.qtrac.eu/pyclassmulti.html)
- set default number of workers to all threads
@BerndDoser
add torchvision.transforms.v2 to allsky
6967b90
@BerndDoser
fix psutil number of workers
e9b162c
@BerndDoser BerndDoser linked an issue Dec 7, 2023 that may be closed by this pull request
Abstract classes SpherinatorDataModule and SpherinatorDataSet #51
Closed
@BerndDoser
move write_catalog from hipster to datamodule
dcadc74
@BerndDoser
fix hipster output path
f7c5cb9
@BerndDoser
DatasetWithMetadata and transformations as class variable
ed2c7df
@BerndDoser
hipster write catalog
e31dadb
@BerndDoser
ShapesDatasetWithMetadata
5eb8c3c
@BerndDoser
load trainer in hipster
15851e6
@BerndDoser
fix transform_processing as instance variable
838e2dc
@BerndDoser
test write catalog and set seed for pytest in general
cde46b6
@BerndDoser
test create images of shapes dataset
43999b0 
remove mixin class design
@BerndDoser
pandas testing csv files for float tolerance
56145b8
@BerndDoser
typo
7944107
@BerndDoser
add missing id in catalog.csv
f3ecae8
@BerndDoser
remove development and sweep scripts
f69d2eb
@BerndDoser
fix missing best rotation value
06a2a3f
@BerndDoser
fix hipster projection
f99341d 
hipster verbosity print level
thumbnail images
@BerndDoser
some links
25d6f0f
@BerndDoser
resize images
77e15e0
@BerndDoser
healpy is not fully reproducible
2b823ba
@BerndDoser
create images and thumbnails for illustris dataset
f5cd905
@BerndDoser
fix allsky import
fbff5cc
@BerndDoser
fix max_order in allsky
dcb7a89
@BerndDoser
create image and thumbnails paths
f1cb1fb
@BerndDoser
center image
62cb971
@BerndDoser
remove wandb from model config file
82f9805
@BerndDoser
Merge branch 'data' of https://github.com/BerndDoser/Spherinator into…
59f2e5c 
… data
@BerndDoser
fix catalog index
f22d322
@BerndDoser
drop test data item 987
c9cf449
@BerndDoser BerndDoser merged commit 91636a2 into HITS-AIN:main Dec 14, 2023
@BerndDoser BerndDoser deleted the data branch December 14, 2023 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@BerndDoser BerndDoser

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Abstract classes SpherinatorDataModule and SpherinatorDataSet Misuse of val_dataloader
1 participant
@BerndDoser