-
-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add CLI App Support #389
Add CLI App Support #389
Changes from all commits
8d3b714
459b3df
e1d8520
fa004fd
c3e6396
946771a
dee064b
16d2978
6af5673
b2da381
802cf5b
e683647
200b906
8dfd566
daf45f6
b47e164
71d2cae
c1ea0b8
7bb59c7
fffd2ec
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -507,8 +507,7 @@ models. There are two primary use cases for Pydantic settings CLI: | |
|
||
By default, the experience is tailored towards use case #1 and builds on the foundations established in [parsing | ||
environment variables](#parsing-environment-variable-values). If your use case primarily falls into #2, you will likely | ||
want to enable [enforcing required arguments at the CLI](#enforce-required-arguments-at-cli) and [nested model default | ||
partial updates](#nested-model-default-partial-updates). | ||
want to enable most of the defaults outlined at the end of [creating CLI applications](#creating-cli-applications). | ||
|
||
### The Basics | ||
|
||
|
@@ -560,19 +559,7 @@ print(Settings().model_dump()) | |
``` | ||
|
||
To enable CLI parsing, we simply set the `cli_parse_args` flag to a valid value, which retains similar conotations as | ||
defined in `argparse`. Alternatively, we can also directly provide the args to parse at time of instantiation: | ||
|
||
```py | ||
from pydantic_settings import BaseSettings | ||
|
||
|
||
class Settings(BaseSettings): | ||
this_foo: str | ||
|
||
|
||
print(Settings(_cli_parse_args=['--this_foo', 'is such a foo']).model_dump()) | ||
#> {'this_foo': 'is such a foo'} | ||
``` | ||
defined in `argparse`. | ||
|
||
Note that a CLI settings source is [**the topmost source**](#field-value-priority) by default unless its [priority value | ||
is customised](#customise-settings-sources): | ||
|
@@ -875,6 +862,95 @@ sys.argv = ['example.py', 'gamma-cmd', '--opt-gamma=hi'] | |
assert get_subcommand(Root()).model_dump() == {'opt_gamma': 'hi'} | ||
``` | ||
|
||
### Creating CLI Applications | ||
|
||
The `CliApp` class provides two utility methods, `CliApp.run` and `CliApp.run_subcommand`, that can be used to run a | ||
Pydantic `BaseSettings`, `BaseModel`, or `pydantic.dataclasses.dataclass` as a CLI application. Primarily, the methods | ||
provide structure for running `cli_cmd` methods associated with models. | ||
|
||
`CliApp.run` can be used in directly providing the `cli_args` to be parsed, and will run the model `cli_cmd` method (if | ||
defined) after instantiation: | ||
|
||
```py | ||
from pydantic_settings import BaseSettings, CliApp | ||
|
||
|
||
class Settings(BaseSettings): | ||
this_foo: str | ||
|
||
def cli_cmd(self) -> None: | ||
# Print the parsed data | ||
print(self.model_dump()) | ||
#> {'this_foo': 'is such a foo'} | ||
|
||
# Update the parsed data showing cli_cmd ran | ||
self.this_foo = 'ran the foo cli cmd' | ||
|
||
|
||
s = CliApp.run(Settings, cli_args=['--this_foo', 'is such a foo']) | ||
print(s.model_dump()) | ||
#> {'this_foo': 'ran the foo cli cmd'} | ||
``` | ||
|
||
Similarly, the `CliApp.run_subcommand` can be used in recursive fashion to run the `cli_cmd` method of a subcommand: | ||
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
```py | ||
from pydantic import BaseModel | ||
|
||
from pydantic_settings import CliApp, CliPositionalArg, CliSubCommand | ||
|
||
|
||
class Init(BaseModel): | ||
directory: CliPositionalArg[str] | ||
|
||
def cli_cmd(self) -> None: | ||
print(f'git init "{self.directory}"') | ||
#> git init "dir" | ||
self.directory = 'ran the git init cli cmd' | ||
|
||
|
||
class Clone(BaseModel): | ||
repository: CliPositionalArg[str] | ||
directory: CliPositionalArg[str] | ||
|
||
def cli_cmd(self) -> None: | ||
print(f'git clone from "{self.repository}" into "{self.directory}"') | ||
self.directory = 'ran the clone cli cmd' | ||
|
||
|
||
class Git(BaseModel): | ||
clone: CliSubCommand[Clone] | ||
init: CliSubCommand[Init] | ||
|
||
def cli_cmd(self) -> None: | ||
CliApp.run_subcommand(self) | ||
|
||
|
||
cmd = CliApp.run(Git, cli_args=['init', 'dir']) | ||
assert cmd.model_dump() == { | ||
'clone': None, | ||
'init': {'directory': 'ran the git init cli cmd'}, | ||
} | ||
``` | ||
|
||
!!! note | ||
Unlike `CliApp.run`, `CliApp.run_subcommand` requires the subcommand model to have a defined `cli_cmd` method. | ||
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
For `BaseModel` and `pydantic.dataclasses.dataclass` types, `CliApp.run` will internally use the following | ||
`BaseSettings` configuration defaults: | ||
|
||
* `alias_generator=AliasGenerator(lambda s: s.replace('_', '-'))` | ||
* `nested_model_default_partial_update=True` | ||
* `case_sensitive=True` | ||
* `cli_hide_none_type=True` | ||
* `cli_avoid_json=True` | ||
* `cli_enforce_required=True` | ||
* `cli_implicit_flags=True` | ||
|
||
!!! note | ||
The alias generator for kebab case does not propagate to subcommands or submodels and will have to be manually set | ||
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
|
||
in these cases. | ||
|
||
### Customizing the CLI Experience | ||
|
||
The below flags can be used to customise the CLI experience to your needs. | ||
|
@@ -1241,7 +1317,7 @@ defined one that specifies the `root_parser` object. | |
import sys | ||
from argparse import ArgumentParser | ||
|
||
from pydantic_settings import BaseSettings, CliSettingsSource | ||
from pydantic_settings import BaseSettings, CliApp, CliSettingsSource | ||
|
||
parser = ArgumentParser() | ||
parser.add_argument('--food', choices=['pear', 'kiwi', 'lime']) | ||
|
@@ -1256,13 +1332,15 @@ cli_settings = CliSettingsSource(Settings, root_parser=parser) | |
|
||
# Parse and load CLI settings from the command line into the settings source. | ||
sys.argv = ['example.py', '--food', 'kiwi', '--name', 'waldo'] | ||
print(Settings(_cli_settings_source=cli_settings(args=True)).model_dump()) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed suggestions of using private param |
||
s = CliApp.run(Settings, cli_settings_source=cli_settings) | ||
print(s.model_dump()) | ||
#> {'name': 'waldo'} | ||
|
||
# Load CLI settings from pre-parsed arguments. i.e., the parsing occurs elsewhere and we | ||
# just need to load the pre-parsed args into the settings source. | ||
parsed_args = parser.parse_args(['--food', 'kiwi', '--name', 'ralph']) | ||
print(Settings(_cli_settings_source=cli_settings(parsed_args=parsed_args)).model_dump()) | ||
s = CliApp.run(Settings, cli_args=parsed_args, cli_settings_source=cli_settings) | ||
print(s.model_dump()) | ||
#> {'name': 'ralph'} | ||
``` | ||
|
||
|
@@ -1281,6 +1359,11 @@ parser methods that can be customised, along with their argparse counterparts (t | |
For a non-argparse parser the parser methods can be set to `None` if not supported. The CLI settings will only raise an | ||
error when connecting to the root parser if a parser method is necessary but set to `None`. | ||
|
||
!!! note | ||
The `formatter_class` is only applied to subcommands. The `CliSettingsSource` never touches or modifies any of the | ||
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
|
||
external parser settings to avoid breaking changes. Since subcommands reside on their own internal parser trees, we | ||
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
kschwab marked this conversation as resolved.
Show resolved
Hide resolved
|
||
can safely apply the `formatter_class` settings without breaking the external parser logic. | ||
|
||
## Secrets | ||
|
||
Placing secret values in files is a common pattern to provide sensitive configuration to an application. | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,14 @@ | ||
from __future__ import annotations as _annotations | ||
|
||
from typing import Any, ClassVar | ||
from argparse import Namespace | ||
from types import SimpleNamespace | ||
from typing import Any, ClassVar, TypeVar | ||
|
||
from pydantic import ConfigDict | ||
from pydantic import AliasGenerator, ConfigDict | ||
from pydantic._internal._config import config_keys | ||
from pydantic._internal._utils import deep_update | ||
from pydantic._internal._signature import _field_name_for_signature | ||
from pydantic._internal._utils import deep_update, is_model_class | ||
from pydantic.dataclasses import is_pydantic_dataclass | ||
from pydantic.main import BaseModel | ||
|
||
from .sources import ( | ||
|
@@ -17,9 +21,14 @@ | |
InitSettingsSource, | ||
PathType, | ||
PydanticBaseSettingsSource, | ||
PydanticModel, | ||
SecretsSettingsSource, | ||
SettingsError, | ||
get_subcommand, | ||
) | ||
|
||
T = TypeVar('T') | ||
|
||
|
||
class SettingsConfigDict(ConfigDict, total=False): | ||
case_sensitive: bool | ||
|
@@ -33,7 +42,6 @@ class SettingsConfigDict(ConfigDict, total=False): | |
env_parse_enums: bool | None | ||
cli_prog_name: str | None | ||
cli_parse_args: bool | list[str] | tuple[str, ...] | None | ||
cli_settings_source: CliSettingsSource[Any] | None | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was a bug. It's not possible to provide |
||
cli_parse_none_str: str | None | ||
cli_hide_none_type: bool | ||
cli_avoid_json: bool | ||
|
@@ -91,7 +99,8 @@ class BaseSettings(BaseModel): | |
All the below attributes can be set via `model_config`. | ||
|
||
Args: | ||
_case_sensitive: Whether environment variables names should be read with case-sensitivity. Defaults to `None`. | ||
_case_sensitive: Whether environment and CLI variable names should be read with case-sensitivity. | ||
Defaults to `None`. | ||
_nested_model_default_partial_update: Whether to allow partial updates on nested model default object fields. | ||
Defaults to `False`. | ||
_env_prefix: Prefix for all environment variables. Defaults to `None`. | ||
|
@@ -345,26 +354,24 @@ def _settings_build_values( | |
file_secret_settings=file_secret_settings, | ||
) + (default_settings,) | ||
if not any([source for source in sources if isinstance(source, CliSettingsSource)]): | ||
if cli_parse_args is not None or cli_settings_source is not None: | ||
cli_settings = ( | ||
CliSettingsSource( | ||
self.__class__, | ||
cli_prog_name=cli_prog_name, | ||
cli_parse_args=cli_parse_args, | ||
cli_parse_none_str=cli_parse_none_str, | ||
cli_hide_none_type=cli_hide_none_type, | ||
cli_avoid_json=cli_avoid_json, | ||
cli_enforce_required=cli_enforce_required, | ||
cli_use_class_docs_for_groups=cli_use_class_docs_for_groups, | ||
cli_exit_on_error=cli_exit_on_error, | ||
cli_prefix=cli_prefix, | ||
cli_flag_prefix_char=cli_flag_prefix_char, | ||
cli_implicit_flags=cli_implicit_flags, | ||
cli_ignore_unknown_args=cli_ignore_unknown_args, | ||
case_sensitive=case_sensitive, | ||
) | ||
if cli_settings_source is None | ||
else cli_settings_source | ||
if isinstance(cli_settings_source, CliSettingsSource): | ||
sources = (cli_settings_source,) + sources | ||
elif cli_parse_args is not None: | ||
cli_settings = CliSettingsSource[Any]( | ||
self.__class__, | ||
cli_prog_name=cli_prog_name, | ||
cli_parse_args=cli_parse_args, | ||
cli_parse_none_str=cli_parse_none_str, | ||
cli_hide_none_type=cli_hide_none_type, | ||
cli_avoid_json=cli_avoid_json, | ||
cli_enforce_required=cli_enforce_required, | ||
cli_use_class_docs_for_groups=cli_use_class_docs_for_groups, | ||
cli_exit_on_error=cli_exit_on_error, | ||
cli_prefix=cli_prefix, | ||
cli_flag_prefix_char=cli_flag_prefix_char, | ||
cli_implicit_flags=cli_implicit_flags, | ||
cli_ignore_unknown_args=cli_ignore_unknown_args, | ||
case_sensitive=case_sensitive, | ||
) | ||
sources = (cli_settings,) + sources | ||
if sources: | ||
|
@@ -401,7 +408,6 @@ def _settings_build_values( | |
env_parse_enums=None, | ||
cli_prog_name=None, | ||
cli_parse_args=None, | ||
cli_settings_source=None, | ||
cli_parse_none_str=None, | ||
cli_hide_none_type=False, | ||
cli_avoid_json=False, | ||
|
@@ -420,3 +426,114 @@ def _settings_build_values( | |
secrets_dir=None, | ||
protected_namespaces=('model_', 'settings_'), | ||
) | ||
|
||
|
||
class CliApp: | ||
""" | ||
A utility class for running Pydantic `BaseSettings`, `BaseModel`, or `pydantic.dataclasses.dataclass` as | ||
CLI applications. | ||
""" | ||
|
||
@staticmethod | ||
def _run_cli_cmd(model: Any, cli_cmd_method_name: str, is_required: bool) -> Any: | ||
if hasattr(type(model), cli_cmd_method_name): | ||
getattr(type(model), cli_cmd_method_name)(model) | ||
elif is_required: | ||
raise SettingsError(f'Error: {type(model).__name__} class is missing {cli_cmd_method_name} entrypoint') | ||
return model | ||
|
||
@staticmethod | ||
def run( | ||
model_cls: type[T], | ||
cli_args: list[str] | Namespace | SimpleNamespace | dict[str, Any] | None = None, | ||
cli_settings_source: CliSettingsSource[Any] | None = None, | ||
cli_exit_on_error: bool | None = None, | ||
cli_cmd_method_name: str = 'cli_cmd', | ||
**model_init_data: Any, | ||
) -> T: | ||
""" | ||
Runs a Pydantic `BaseSettings`, `BaseModel`, or `pydantic.dataclasses.dataclass` as a CLI application. | ||
Running a model as a CLI application requires the `cli_cmd` method to be defined in the model class. | ||
|
||
Args: | ||
model_cls: The model class to run as a CLI application. | ||
cli_args: The list of CLI arguments to parse. If `cli_settings_source` is specified, this may | ||
also be a namespace or dictionary of pre-parsed CLI arguments. Defaults to `sys.argv[1:]`. | ||
cli_settings_source: Override the default CLI settings source with a user defined instance. | ||
Defaults to `None`. | ||
cli_exit_on_error: Determines whether this function exits on error. If model is subclass of | ||
`BaseSettings`, defaults to BaseSettings `cli_exit_on_error` value. Otherwise, defaults to | ||
`True`. | ||
cli_cmd_method_name: The CLI command method name to run. Defaults to "cli_cmd". | ||
model_init_data: The model init data. | ||
|
||
Returns: | ||
The ran instance of model. | ||
|
||
Raises: | ||
SettingsError: If model_cls is not subclass of `BaseModel` or `pydantic.dataclasses.dataclass`. | ||
SettingsError: If model_cls does not have a `cli_cmd` entrypoint defined. | ||
""" | ||
|
||
if not (is_pydantic_dataclass(model_cls) or is_model_class(model_cls)): | ||
raise SettingsError( | ||
f'Error: {model_cls.__name__} is not subclass of BaseModel or pydantic.dataclasses.dataclass' | ||
) | ||
|
||
cli_settings = None | ||
cli_parse_args = True if cli_args is None else cli_args | ||
if cli_settings_source is not None: | ||
if isinstance(cli_parse_args, (Namespace, SimpleNamespace, dict)): | ||
cli_settings = cli_settings_source(parsed_args=cli_parse_args) | ||
else: | ||
cli_settings = cli_settings_source(args=cli_parse_args) | ||
elif isinstance(cli_parse_args, (Namespace, SimpleNamespace, dict)): | ||
raise SettingsError('Error: `cli_args` must be list[str] or None when `cli_settings_source` is not used') | ||
|
||
model_init_data['_cli_parse_args'] = cli_parse_args | ||
model_init_data['_cli_exit_on_error'] = cli_exit_on_error | ||
model_init_data['_cli_settings_source'] = cli_settings | ||
if not issubclass(model_cls, BaseSettings): | ||
|
||
class CliAppBaseSettings(BaseSettings, model_cls): # type: ignore | ||
model_config = SettingsConfigDict( | ||
alias_generator=AliasGenerator(lambda s: s.replace('_', '-')), | ||
nested_model_default_partial_update=True, | ||
case_sensitive=True, | ||
cli_hide_none_type=True, | ||
cli_avoid_json=True, | ||
cli_enforce_required=True, | ||
cli_implicit_flags=True, | ||
) | ||
|
||
model = CliAppBaseSettings(**model_init_data) | ||
model_init_data = {} | ||
for field_name, field_info in model.model_fields.items(): | ||
model_init_data[_field_name_for_signature(field_name, field_info)] = getattr(model, field_name) | ||
|
||
return CliApp._run_cli_cmd(model_cls(**model_init_data), cli_cmd_method_name, is_required=False) | ||
|
||
@staticmethod | ||
def run_subcommand( | ||
model: PydanticModel, cli_exit_on_error: bool | None = None, cli_cmd_method_name: str = 'cli_cmd' | ||
) -> PydanticModel: | ||
""" | ||
Runs the model subcommand. Running a model subcommand requires the `cli_cmd` method to be defined in | ||
the nested model subcommand class. | ||
|
||
Args: | ||
model: The model to run the subcommand from. | ||
cli_exit_on_error: Determines whether this function exits with error if no subcommand is found. | ||
Defaults to model_config `cli_exit_on_error` value if set. Otherwise, defaults to `True`. | ||
cli_cmd_method_name: The CLI command method name to run. Defaults to "cli_cmd". | ||
|
||
Returns: | ||
The ran subcommand model. | ||
|
||
Raises: | ||
SystemExit: When no subcommand is found and cli_exit_on_error=`True` (the default). | ||
SettingsError: When no subcommand is found and cli_exit_on_error=`False`. | ||
""" | ||
|
||
subcommand = get_subcommand(model, is_required=True, cli_exit_on_error=cli_exit_on_error) | ||
return CliApp._run_cli_cmd(subcommand, cli_cmd_method_name, is_required=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed suggestion of using private param
_cli_parse_args
. The "formal" equivalent is now: