New experiment config #3138

liuzhe-lz · 2020-11-29T22:46:45Z

This PR defines the new experiment config schema.

Example Usage

Following code snippet will generate a valid config.yaml file:

from ruamel import yaml
from nni.experiment.config import ExperimentConfig, AlgorithmConfig, LocalConfig, convert

config = ExperimentConfig(
    experiment_name = 'MNIST',
    trial_concurrency = 1,
    max_experiment_duration = '10m',
    max_trial_number = 10,
    search_space_file = 'search_space.json',
    tuner = AlgorithmConfig(name='TPE', class_args={'optimize_mode': 'maximize'}),
    trial_command = 'python3 mnist.py',
    trial_code_directory = '.',
    training_service = LocalConfig(use_active_gpu=True)
)

config = convert.to_old_yaml(config)
yaml.dump(config, open('config.yaml', 'w'))

Or alternatively:

from ruamel import yaml
from nni.experiment.config import ExperimentConfig, convert

config = ExperimentConfig('local')
config.trial_concurrency = 1
config.max_experiment_duration = '10m'
config.max_trial_number = 10
config.search_space = {
    "dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]},
    "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}
}
config.tuner = AlgorithmConfig(name='TPE', class_args={'optimize_mode': 'maximize'})
config.trial_command = 'python3 mnist.py'
config.trial_code_directory = '.'
config.training_service.use_active_gpu = True

config = convert.to_old_yaml(config)
yaml.dump(config, open('config.yaml', 'w'))

Design Notes

The schema should be expressible in both YAML/JSON object and Python dataclass.
In YAML/JSON the keys are expected to be camelCase and in Python they should be snake_case.
This technically is not a problem since Config.load() and Config.json() have routines to do the convert.

I think the consistency with old schema is not a case. Because consistence is to help migration, while we should not push exist users to migrate at all.
The most important goal of this refactor is to reduce users' learning cost. And for exist users, to not learn the cost is zero.
I believe we will continue to support old schema for very long time, maybe until NNI v3.0. So it basically benefits no body to push the migration unless a user is unsatisfied old schema and proactively want a new one.
And in the other hand, the more difference between two schema, the smaller possibility NNI may fail to detect schema version.

Taking Yuge's advice, the config is divided into platform-independent part and nested training service config.
Trial config is no longer nested because I can't find a reason to do so. While tuner/assessor/advisor config is still nested because the same format may repeats for more than one algorithms.
There is a problem that whether training service config should share the same key. Using same key is definitely better in Python, but using "localCofing" / "remoteConfig" may looks better in YAML.
The problem might affect heterogeneous training service. If training services use same key "training service config", it will be a object/dict for heterogeneous. And if they use different keys, there will be multiple training service config objects.

liuzhe-lz · 2020-11-29T22:47:01Z

reference: https://github.com/microsoft/nni/blob/b452c32abf70567040f8cc375443890ead9940e2/docs/en_US/reference/experiment_config.rst

onenote: https://microsoftapc.sharepoint.com/teams/NeuralNetworkIntelligence/_layouts/15/Doc.aspx?sourcedoc={2d50c14e-4fad-4b5f-8d55-f181ec62a94e}&action=edit&wd=target%28Feature%20Design.one%7C66695e35-12fe-47c2-948c-e5d9926583a9%2FExperiment%20Config%20Schema%7C6a34742d-3e0e-4f5b-b967-061b64dc5d73%2F%29&wdorigin=703

SparkSnail · 2020-12-11T07:22:07Z

Does this pr support heterogeneousConfig?

ultmaster · 2020-12-12T04:31:49Z