Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Switch to version 1.0 of config file format, fix #685 #345 #748 #750

Merged
merged 183 commits into from
May 5, 2024
Merged
Changes from 1 commit
Commits
Show all changes
183 commits
Select commit Hold shift + click to select a range
476b9da
WIP: Add src/vak/config/dataset.py
NickleDave May 1, 2024
a625bb7
Add module-level docstring + type annotations in src/vak/config/parse.py
NickleDave May 1, 2024
272f67d
WIP: Fix how cli.prep adds dataset path to toml config file
NickleDave May 1, 2024
fa35def
Change table names in src/vak/config/valid.toml
NickleDave May 1, 2024
988c150
Rename section -> table in config/parse.py
NickleDave May 1, 2024
72e11e8
In cli/prep change 'section' -> 'table' and lowercase table names
NickleDave May 1, 2024
7181f71
In config/config.py, change 'section' -> 'table' and lowercase table …
NickleDave May 1, 2024
9d4467c
Change '[PREP]' -> '[vak.prep]' in config/prep.py
NickleDave May 1, 2024
b5a6413
WIP: Change table names in config files in tests/data_for_tests/configs
NickleDave May 1, 2024
27f0596
Make tomlkit a dependency in pyproject.toml, drop toml
NickleDave May 1, 2024
62fe4f4
Change config/parse.py to use tomlkit
NickleDave May 1, 2024
3253a4e
Update example configs in doc/toml/
NickleDave May 1, 2024
fb70733
Add link to example config files in docs, in error messages in config…
NickleDave May 1, 2024
b653138
Remove 'spect_params' from REQUIRED_OPTIONS in config/parse.py, this …
NickleDave May 1, 2024
83eddf0
Rename 'config_toml' -> 'config_dict' in config/parse.py
NickleDave May 1, 2024
1160a6b
Fix function _validate_tables_arg_convert_list in config/parse.py
NickleDave May 1, 2024
9796624
Fix error message formatting in src/vak/config/validators.py
NickleDave May 1, 2024
60f7b02
Add ModelConfig class to config/model.py, add type annotations, fix c…
NickleDave May 1, 2024
910c5df
Fixup fixing config_from_toml_dict to look in specific section
NickleDave May 1, 2024
32995bf
Rewrite config/eval.py with 'modern' attrs
NickleDave May 1, 2024
f775749
Fixup rewrite config/eval with 'modern attrs
NickleDave May 1, 2024
060abaa
Rewrite config/learncurve.py with 'modern' attrs
NickleDave May 1, 2024
71e89bc
Rewrite config/predict.py with 'modern' attrs
NickleDave May 1, 2024
f6cf945
Rewrite config/prep.py with 'modern' attrs
NickleDave May 1, 2024
9798707
Rewrite config/train.py with 'modern' attrs
NickleDave May 1, 2024
9c0d301
Rename Dataset -> DatasetConfig in config/dataset.py
NickleDave May 1, 2024
0de12da
Add are_table_options_valid to config/validators.py, will be used by …
NickleDave May 1, 2024
2c272bc
WIP: Add from_config_dict classmethod to EvalConfig
NickleDave May 1, 2024
53447f5
WIP: Add tests/test_config/test_dataset.py
NickleDave May 1, 2024
a6cb674
Make fixes to ModelConfig class, fix circular imports in config/model…
NickleDave May 1, 2024
a039dd9
Write tests in tests/test_config/test_dataset.py
NickleDave May 1, 2024
d6588f0
Use tomlkit not toml in cli/prep.py
NickleDave May 2, 2024
53ef623
Use tomlkit in tests/fixtures/annot.py
NickleDave May 2, 2024
1f80c19
Use tomlkit in tests/scripts/vaktestdata/configs.py
NickleDave May 2, 2024
cd2d0dd
Use tomlkit in tests/scripts/vaktestdata/source_files.py
NickleDave May 2, 2024
ad1bbf3
Use tomlkit in tests/test_config/test_validators.py
NickleDave May 2, 2024
249cf16
Remove spect_params attribute from Config in config/config.py, fix cl…
NickleDave May 2, 2024
6654a06
Reorder attributes, fix typo in docstring of DatasetConfig
NickleDave May 2, 2024
b2c8652
Rewrite config/parse.py assuming config classes have from_config_dict…
NickleDave May 2, 2024
0680481
Rename `table` -> `table_name` in a couple validators in config/valid…
NickleDave May 2, 2024
9cbf58a
Remove use of config.model.config_from_toml_path in cli/eval.py
NickleDave May 2, 2024
eeef546
Remove use of config.model.config_from_toml_path in cli/learncurve.py
NickleDave May 2, 2024
0a5eb96
Remove use of config.model.config_from_toml_path in cli/predict.py
NickleDave May 2, 2024
f8ebc20
Remove use of config.model.config_from_toml_path in cli/train.py
NickleDave May 2, 2024
cebd8d8
Remove functions from config/model.py: config_from_toml_path and conf…
NickleDave May 2, 2024
7469f8d
Add `to_dict` method to ModelConfig
NickleDave May 2, 2024
c0c11e4
Use to_dict() method of ModelConfig class in cli functions
NickleDave May 2, 2024
e87b5ed
Fix how we get labelset from config in tests/fixtures/annot.py
NickleDave May 2, 2024
07dd40e
WIP: Clean up / rewrite tests/fixtures/config.py
NickleDave May 2, 2024
22aa552
Fix model tables in tests/data_for_tests/configs
NickleDave May 2, 2024
10823a5
Finish unit tests in tests/test_config/test_model.py
NickleDave May 2, 2024
2216f4b
Fix model tables in doc/toml
NickleDave May 2, 2024
4c38198
Rename data_for_tests/configs/invalid_option_config.toml -> invalid_k…
NickleDave May 2, 2024
9f8b222
Rename are_options_valid/are_table_options_valid -> are_keys_valid/ar…
NickleDave May 2, 2024
cdf7495
Rename two fixtures in fixtures/config.py: invalid_section_config_pat…
NickleDave May 2, 2024
90f401f
Fix validator names in config/parse.py, rename TABLE_CLASSES constant…
NickleDave May 2, 2024
67ec4a9
Rename config/valid.toml -> valid-version-1.0.toml, fix how model tab…
NickleDave May 2, 2024
7288933
Fix VALID_TOML_PATH in config/validators.py after renaming config/val…
NickleDave May 2, 2024
4f72b0c
Import config classes in vak/config/__init__.py
NickleDave May 2, 2024
bd0a055
Add _tomlkit_to_popo to tests/fixtures/config.py so we operate on dic…
NickleDave May 2, 2024
e359376
Add _tomlkit_to_popo to config/parse.py so we operate on dicts not to…
NickleDave May 2, 2024
0ff5860
Finish rewriting tests for tests/test_config/test_prep.py
NickleDave May 2, 2024
3e5f1d2
Rewrite EvalConfig with from_config_dict method
NickleDave May 2, 2024
aeee2b6
Rewrite LearncurveConfig with from_config_dict method
NickleDave May 2, 2024
6ab3a5f
Rewrite PredictConfig with from_config_dict method
NickleDave May 2, 2024
4b4fedf
Rewrite PrepConfig with from_config_dict method
NickleDave May 2, 2024
77e38e6
Rewrite TrainConfig with from_config_dict method
NickleDave May 2, 2024
f3e37ae
Remove functions from config/parse.py
NickleDave May 2, 2024
2b598d0
Rename config/parse.py -> config/load.py
NickleDave May 2, 2024
d175436
Make functions in config/parse.py into classmethods on Config class
NickleDave May 2, 2024
323c238
Use config.Config.from_toml_path everywhere instead of config.parse.f…
NickleDave May 2, 2024
500543e
Make fixes in Config classmethods
NickleDave May 3, 2024
7ebd92e
Change load._load_toml_from_path again so that it returns config_dict…
NickleDave May 3, 2024
acf4cc3
Add docstring to are_tables_valid in config/validators.py
NickleDave May 3, 2024
42e92c7
Lowercase config table names in tests/scripts/vaktestdata/configs.py
NickleDave May 3, 2024
61ea4e4
In tests/scripts/vaktestdata/source_files.py, change cfg.spect_params…
NickleDave May 3, 2024
dc19398
in test_cli/test_prep.py, call vak.config.load not vak.config.parse
NickleDave May 3, 2024
95e424c
Fix how we instantiate DatasetConfig and ModelConfig in EvalConfig.fr…
NickleDave May 3, 2024
5481ec8
Fix how we instantiate DatasetConfig and ModelConfig in PredictConfig…
NickleDave May 3, 2024
1733166
Fix how we instantiate DatasetConfig and ModelConfig in TrainConfig.f…
NickleDave May 3, 2024
d729abd
Fix how we instantiate DatasetConfig and ModelConfig in LearncurveCon…
NickleDave May 3, 2024
9185eb9
Remove brekapoint in src/vak/config/model.py
NickleDave May 3, 2024
3341b08
Fix wrong variable name so we save configs correctly in tests/scripts…
NickleDave May 3, 2024
b050989
Fix how we re-write configs, in tests/scripts/vaktestdata/configs.py
NickleDave May 3, 2024
f6fdec6
Add model and dataset tables to get those keys in top-level tables, i…
NickleDave May 3, 2024
8d49aef
Change cfg.table.dataset_path -> cfg.table.dataset.path in vak/cli mo…
NickleDave May 3, 2024
2798282
Get tests passing for tests/test_config/test_eval.py
NickleDave May 3, 2024
0f6cc5a
Clean up tests/test_config/test_eval.py
NickleDave May 3, 2024
311287f
Get tests passing in tests/test_config/test_predict.py
NickleDave May 3, 2024
5c88bb0
Fix how we access config_toml in tests/scripts/vaktestdata/configs.py…
NickleDave May 3, 2024
5db3d2f
Add pytest.mark.parametrize to tests/test_config/test_learncurve.py
NickleDave May 3, 2024
3180d48
Rewrite tests in tests/test_config/test_train.py
NickleDave May 3, 2024
7c2a1f0
Rewrite tests in tests/test_config/test_config.py
NickleDave May 3, 2024
18b6dc2
Add unit test to tests/test_config/test_model.py
NickleDave May 3, 2024
a4c06d5
Add unit test for exceptions in tests/test_config/test_eval.py
NickleDave May 3, 2024
05032d6
Fix 'cfg.spect_params' -> 'cfg.prep.spect_params' in src/vak/cli/pred…
NickleDave May 3, 2024
d96d782
Add unit test for exceptions in tests/test_config/test_learncurve.py
NickleDave May 3, 2024
3927c5b
Add unit test for exceptions in tests/test_config/test_train.py
NickleDave May 3, 2024
1996df2
Add more test cases to TestEvalConfig.test_from_config_dict_raises
NickleDave May 3, 2024
30a31f8
Add more test cases to TestLearncurveConfig.test_from_config_dict_raises
NickleDave May 3, 2024
836e8ed
Add unit test for exceptions in tests/test_config/test_predict.py
NickleDave May 3, 2024
8e2cc9f
Add two unit tests that PrepConfig raises expected exceptions
NickleDave May 3, 2024
35c3732
Fix/add unit tests in tests/test_config/test_config.py
NickleDave May 3, 2024
e0abf23
Change order of parameters for Config.from_config_dict, make toml_pat…
NickleDave May 3, 2024
e585091
Fix/add unit tests in tests/fixtures/config.py
NickleDave May 3, 2024
ffd0a4e
Fix/add unit tests in tests/fixtures/config.py
NickleDave May 3, 2024
63ee0ee
Rename test_config/test_parse.py -> test_load.py, fix/rewrite tests
NickleDave May 3, 2024
f081359
Fix tests in tests/test_config/test_spect_params.py
NickleDave May 3, 2024
dcc98a8
Make fixups in tests/test_config
NickleDave May 3, 2024
82dd4f3
Apply fixes from linter
NickleDave May 3, 2024
a03b3b1
Make more linting fixes
NickleDave May 3, 2024
9b31067
Speed up install in nox session 'lint', only install linting tools
NickleDave May 3, 2024
ee9c4e5
Change names 'section'/'option' -> 'table'/'key' in tests
NickleDave May 3, 2024
9466d6c
Fix tests in tests/test_cli/test_eval.py
NickleDave May 3, 2024
0206455
Finish fixing cli tests, fix renaming
NickleDave May 3, 2024
2ed3763
Fix how we get 'path' from 'dataset' table in configs, in tests/fixtu…
NickleDave May 3, 2024
fca4ba9
Fix how we get 'path' from 'dataset' table in configs, in tests/fixtu…
NickleDave May 3, 2024
d71d773
Change .dataset_path -> .dataset.path in tests/
NickleDave May 3, 2024
4a5122d
Fix how we get model config and rename config attribute .dataset_path…
NickleDave May 3, 2024
b8b902c
In tests/, fixup change .dataset_path -> .dataset.path, use model.nam…
NickleDave May 3, 2024
631e403
Fix fixture specific_config_toml_path in fixtures/config.py to handle…
NickleDave May 3, 2024
f717e00
Fix how we change ['dataset']['path'] value in tests/test_eval/test_f…
NickleDave May 3, 2024
ac1a7e7
Fix how we change ['dataset']['path'] value in config in several tests
NickleDave May 3, 2024
e4ac360
Use ModelConfig attribute name where needed in tests/test_learncurve/…
NickleDave May 3, 2024
f542c47
In tests, replace calls to vak.config.model.config_from_toml_path wit…
NickleDave May 3, 2024
0ceadfe
Change cfg.spect_params -> cfg.prep.spect_params in tests
NickleDave May 3, 2024
315294a
Fix cfg.predict -> cfg.predict.dataset.path in tests/test_predict/tes…
NickleDave May 3, 2024
7ad30b0
Fix constant LABELSET_NOTMAT in fixtures/annot.py so it is a list of …
NickleDave May 3, 2024
937192d
Fix cfg.learncurve -> cfg.learncurve.dataset.path in tests/test_prep/…
NickleDave May 3, 2024
ef9cd86
Fix cfg.learncurve -> cfg.learncurve.dataset.path in tests/test_prep/…
NickleDave May 4, 2024
ee4a9a9
Cast pathlib to str before adding to tomldoc, in tests/test_train/
NickleDave May 4, 2024
881f4f8
Change transform/dataset params keys in data_for_tests/configs to a d…
NickleDave May 4, 2024
eda99eb
Add `params` attribute to DatasetConfig
NickleDave May 4, 2024
513046f
Change transform/dataset params keys in doc/toml/ to a dataset table …
NickleDave May 4, 2024
5317dff
Rewrite vak/config/model.py method 'to_dict' as 'asdict', using attrs…
NickleDave May 4, 2024
a1f0e0a
Add asdict method to DatasetConfig class, like ModelConfig.asdict
NickleDave May 4, 2024
bd0d050
Fix calls to model.to_dict() -> model.asdict()
NickleDave May 4, 2024
ba788c6
Add unit tests for DatasetConfig.asdict
NickleDave May 4, 2024
0738b77
Add unit tests for ModelConfig.asdict
NickleDave May 4, 2024
3df6a5c
Add an assertion in tests/test_config/test_dataset.py
NickleDave May 4, 2024
9e2dab9
Remove transform params and dataset_params from EvalConfig, will just…
NickleDave May 4, 2024
1b70d54
Remove dataset/transform_params key-value pairs in valid-version-1.0.…
NickleDave May 4, 2024
48830d4
Remove train/val/dataset/transform_params from TrainConfig, will use …
NickleDave May 4, 2024
ad05a98
Remove train/val/dataset/transform_params from PredictConfig, will us…
NickleDave May 4, 2024
7f93b74
Revise transforms.defaults.frame_classification.TrainItemTransform an…
NickleDave May 4, 2024
7152657
Make vak.table.dataset.params into an in-line table in toml files in …
NickleDave May 4, 2024
149d6eb
Fix attribute name in frame_classification.TrainItemTransform.__init_…
NickleDave May 4, 2024
21f7971
Rewrite datasets.frame_classification.WindowDataset to require item_t…
NickleDave May 4, 2024
be8e693
Rewrite datasets.frame_classification.FramesDataset to make item_tran…
NickleDave May 4, 2024
558bb14
Rewrite src/vak/train/frame_classification.py: remove params model_na…
NickleDave May 4, 2024
b54be05
Rewrite src/vak/train/_train.py: remove params model_name, train/val_…
NickleDave May 4, 2024
88f8b49
Rewrite vak/cli/train.py to call train._train.train with just model_c…
NickleDave May 4, 2024
a8721cc
Fix how we unpack batch in training_step method of FrameClassificatio…
NickleDave May 4, 2024
ad5d98b
Change transform_kwargs parameter of transforms.defaults.parametric_u…
NickleDave May 5, 2024
43c525f
Change transform_kwargs parameter of transforms.defaults.frame_classi…
NickleDave May 5, 2024
9e003fc
Change DatasetConfig.params attribute to default to empty dict, so we…
NickleDave May 5, 2024
0b5bb27
Fix DatasetConfig.from_config_dict method to not use dict.get method,…
NickleDave May 5, 2024
b7ca332
Modify transforms.defaults.get so that transform_kwargs is None by de…
NickleDave May 5, 2024
bebaa3f
Rewrite src/vak/train/parametric_umap.py to use model_config and data…
NickleDave May 5, 2024
8f6ef37
Rewrite vak/eval/frame_classification.py to use model_config and data…
NickleDave May 5, 2024
9768b5d
Rewrite vak/eval/parametric_umap.py to use model_config and dataset_c…
NickleDave May 5, 2024
519f7d4
Rewrite vak/eval/eval_.py to use model_config and dataset_config para…
NickleDave May 5, 2024
e68ad49
Rewrite cli.eval to pass model_config and dataset_config into eval_mo…
NickleDave May 5, 2024
dca625f
Unpack dataset_config[params] with ** inside trak/frame_classificatio…
NickleDave May 5, 2024
43d1a99
Rewrite vak/learncurve/frame_classification.py to use model_config an…
NickleDave May 5, 2024
af971a0
Rewrite vak/learncurve/learncurve.py to use model_config and dataset_…
NickleDave May 5, 2024
2257144
Rewrite cli.learncurve to pass model_config and dataset_config into l…
NickleDave May 5, 2024
cd60eab
Rewrite vak/predict/frame_classification.py to use model_config and d…
NickleDave May 5, 2024
571a9c6
Rewrite vak/predict/parametric_umap.py to use model_config and datase…
NickleDave May 5, 2024
aba3353
Rewrite vak/predict/predict.py to use model_config and dataset_config…
NickleDave May 5, 2024
3e237b4
Fix dataset_path -> dataset_config[path] and add missing variable mod…
NickleDave May 5, 2024
9eb7c0c
Fix dataset_path -> dataset_config[path] and add missing variable mod…
NickleDave May 5, 2024
32d9604
Rewrite vak/cli/predict.py to use model_config and dataset_config par…
NickleDave May 5, 2024
87e1a79
Remove non-existent dataset_params variable in vak/predict/frame_clas…
NickleDave May 5, 2024
bfb4a9b
Fix unit tests for DatasetConfig to test 'params' attribute gets hand…
NickleDave May 5, 2024
d70886a
Remove train/val_dataset_params and train/val_transform_params from t…
NickleDave May 5, 2024
401f1a3
Use DatasetConfig.params attribute where we need to in tests/test_dat…
NickleDave May 5, 2024
0ae33a8
Fix method name ModelConfig.to_dict -> asdict in tests/
NickleDave May 5, 2024
e033b25
In tests for eval/learncurve/predict/train, use model_config and data…
NickleDave May 5, 2024
9d7ec3e
Fix use of default transform and dataset.params attribute in test_mod…
NickleDave May 5, 2024
33f578b
Fix config snippets in docs
NickleDave May 5, 2024
84c316b
Apply linting to src/
NickleDave May 5, 2024
bc0b487
Raise 'from e' with errors in eval/predict/train/frame_classification…
NickleDave May 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use tomlkit not toml in cli/prep.py
NickleDave committed May 2, 2024
commit d6588f0599558509d58162b6ec20e4a1320b6acf
4 changes: 2 additions & 2 deletions src/vak/cli/prep.py
Original file line number Diff line number Diff line change
@@ -5,7 +5,7 @@
import shutil
import warnings

import toml
import tomlkit

from .. import config
from .. import prep as prep_module
@@ -145,7 +145,7 @@ def prep(toml_path):
config_toml[table]["dataset"]["path"] = str(dataset_path)

with toml_path.open("w") as fp:
toml.dump(config_toml, fp)
tomlkit.dump(config_toml, fp)

# lastly, copy config to dataset directory root
shutil.copy(src=toml_path, dst=dataset_path)