[feature request] nested config option for inheritance

[Ben-Epstein]

Really amazing tool, thanks for building it! I'm leveraging the yaml_config, and i'm wondering if there's support for having nested values in a config for overriding purposes.

Imagine, for example, you have multiple typer apps (say, generate_data, and label_data), with some shared variables, (say input_file_name, label_names), for example.

@app.command()
def generate_data(label_names: ..., input_data: ...):
    ...

@app.command()
def label_data(label_names: ..., input_data: ...):
    ...

I could imagine a yaml file like so

label_names
    - label1
    - label2

generate_data:
    input_data: generator.csv

label_data:
    input_data: generator_output.csv

Such that a single config can be shared, and you can have overrides for specific typer apps. Curious your thoughts!

[maxb2]

Thanks for the kind words! I've made an app with a nested config before I made this library. I think that would be a great feature to add to this. I'm thinking it would look something like this:

@app.command()
@nested_config # uses function name
@use_yaml_config()
def generate_data(label_names: ..., input_data: ...):
    ...

@app.command()
@nested_config(name="label-data") # optionally change the name
@use_yaml_config()
def label_data(label_names: ..., input_data: ...):
    ...

Where the nested config just subsets the data (and would really just be a wrapper for loader_transformer). I'd have to think more about the shared variables section though.

[Ben-Epstein]

@maxb2 yea this is perfect, exactly what i was thinking. Can I help you at all in implementation? I'd love to use this for a current project

[Ben-Epstein]

@maxb2 in fact I'm not sure if you even need another decorator. I think this itself would be ideal

@use_yaml_config(override_key="") # defaults to the function name

[maxb2]

@maxb2 in fact I'm not sure if you even need another decorator. I think this itself would be ideal

@use_yaml_config(override_key="") # defaults to the function name

I was thinking the same thing, but it doesn't solve the shared variables section. However, you could use yaml anchors to solve this in your config rather than in this library. For example:

label_names: &label_names
    - label1
    - label2

generate_data:
    input_data: generator.csv
    label_names: *label_names

label_data:
    input_data: generator_output.csv
    label_names: *label_names

I will implement the override_key for a simple subset of the config and let you merge sections in the structure of your document, or let you write your own loader for something more complicated (I'll provide an example).

On a separate note, I was working on a minimal example for you and found that setting the list argument in the config does not work with typer-config.

[Ben-Epstein]

I was thinking the same thing, but it doesn't solve the shared variables section

I've implemented a simple version of this (but specifically for yaml only, so not worth PRing since it's not global), but it should be pretty easily portable to your code. Here's the way I did it.

def yaml_loader(param_value: str, override_key: str = "") -> dict:
    """..."""
    with open(param_value, "r", encoding="utf-8") as file:
        conf: dict = yaml.safe_load(file)
    if conf.get(override_key) and isinstance(conf[override_key], dict):
        conf |= conf[override_key]
    return conf

and I simply pass it in like so

def use_yaml_config(
    override_key: str,
    param_name: str = "config",
    param_help: str = "Configuration file.",
) -> Callable:
    """..."""
    loader = functools.partial(yaml_loader, override_key=override_key)
    yaml_conf_callback: Callable = callbacks.conf_callback_factory(
        loaders.loader_transformer(loader, loader_conditional=lambda param_value: param_value)
    )
    return decorators.use_config(callback=yaml_conf_callback, param_name=param_name, param_help=param_help)

It's not clever, and doesn't read the function name by default. But I think this would be the simplest implementation. Because you can pass this all the way to the top, since you just have a dictionary, and can check for an override key.

i'm using it like this

@use_yaml_config(override_key="label_data")
def label_data(...)

Wdyt?

[maxb2]

I'm actually doing something similar with @use_ini_config() since INI requires a top level section. I'll just make an optional section on all the other default decorators to make it consistent.

[maxb2]

@Ben-Epstein see 1.3.0 for this feature!

On a related note, if you would like to write an example for the docs using yaml anchors for shared variables once you figure out your project, please do and open a PR!

[Ben-Epstein]

@maxb2 this is great, thanks! I checked out your PR, but it doesn't quite enable overrides. It looks like it will only look at the section that you specify.

So if I have a function like so

def foo(a: int, b: int):
   ...

and my config is

a: 5
foo:
  b: 6

this will fail, and say missing option a, am I reading that implementation correctly?

[maxb2]

Yes, you should use yaml anchors to do that (works out of the box):

a: &a 
  5 
foo:
  a: *a
  b: 6
bar:
  a: *a
  b: 7

Or, write a transformer and create a decorator:

def your_config_transformer(config: ConfigDict) -> ConfigDict:
    # does whatever it needs to the whole dictionary
    ...

callback = conf_callback_factory(
        loader_transformer(
            yaml_loader,
            loader_conditional=lambda param_value: param_value,
            config_transformer=your_config_transformer,
        )
    )

your_decorator = use_config(callback)

@app.command()
@your_decorator
def command(...):
    ...

I tried to write this library in a way that you can build whatever you need with functional composition.