Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saving ConfigDict to File -- Using ABSL #19

Open
danielkelshaw opened this issue Dec 21, 2022 · 1 comment
Open

Saving ConfigDict to File -- Using ABSL #19

danielkelshaw opened this issue Dec 21, 2022 · 1 comment

Comments

@danielkelshaw
Copy link

The Problem:

I'm currently using the following pattern to load a config from dictionary

from absl import app, flags
from ml_collections import config_flags

from .config import MY_CONFIG_DICT

FLAGS = flags.FLAGS

_CONFIG = config_flags.DEFINE_config_from_dict('config', MY_CONFIG_DICT)

# ...

absl allows me to alter the config from the command line - this is great for running experiments with different parameters. However, if I want to re-use the same configuration for another run, there doesn't seem to be a de-facto way to do this.

For example, I want to be able to run:

python -m src.experiment --config.value1 10 --config.value2 20

and then be able to re-use this config without having to remember to flags.

How can we save and re-load a ConfigDict instance when using absl flags?

My Approach:

My current approach to this is:

In the experiment script:

  1. Load config into flags as above.
  2. Save the updated ConfigDict to a .yml file.

In a postprocessing script:

  1. Load the original config using the same method.
  2. Provide an additional flag which takes a path to the saved .yml file.
  3. Update the original config as follows:
# postprocessing.py

from absl import flags
from ml_collections import config_dict, config_flags

from .config import MY_CONFIG_DICT

FLAGS = flags.FLAGS

_CONFIG = config_flags.DEFINE_config_from_dict('config', MY_CONFIG_DICT)
_LOAD_CONFIG = flags.DEFINE_string('load_config', None, '.yml file from which to update config.')

def update_config():

    def _update_config_dict(a, b) -> config_dict.ConfigDict:

        da = a.to_dict(preserve_field_references=True)
        db = b.to_dict(preserve_field_references=True)

        return config_dict.ConfigDict(da | db)

    with open(FLAGS.load_config, 'r') as f:
        config_from_yaml = yaml.load(f, Loader=yaml.UnsafeLoader)

    FLAGS.config = _update_config_dict(FLAGS.config, config_from_yaml)

def main(_):

    update_config()

    # ...

While this works it doesn't seem a great approach.

Is there any standard way to accomplish this - or should I be dealing with updated configs in a different manner?

@Chutlhu
Copy link

Chutlhu commented Oct 7, 2024

Thank you! I was searching for the same solution.
I think the following line is not necessary

        db = b.to_dict(preserve_field_references=True)

since b is already a dictionary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants