[UI bug] Missing configuration fields in form

[kimwnasptd]

/kind bug

The new Katib UI's form for submitting Experiment CRs is missing the following fields, which are present in the current UI's form:

• ResumePolicy Add missing form fields to new UI #1463
• EarlyStopping Add missing form fields to new UI #1463
• TrialParameters Add missing form fields to new UI #1463

The most important one is the TrialParameters since without it the user cannot submit an Experiment by only using the form.

admission webhook "validating.experiment.katib.kubeflow.org" denied the request: spec.trialTemplate.trialParameters must be specified

/cc @andreyvelich

[kimwnasptd]

I'll continue working on this now that the first iteration of our new UI #1421 is merged.

@andreyvelich @gaocegege @johnugeorge I see that the current UI has a way to automatically detect the Trial parameters. How exactly does this work? Does the UI parse the string content from the provided ConfigMaps and detects the ${trialParameters.<name>}?

Also @andreyvelich could you also add this issue to our Kanban board?

[kimwnasptd]

/assign @kimwnasptd

[andreyvelich]

I'll continue working on this now that the first iteration of our new UI #1421 is merged.

Thanks @kimwnasptd!

@andreyvelich @gaocegege @johnugeorge I see that the current UI has a way to automatically detect the Trial parameters. How exactly does this work? Does the UI parse the string content from the provided ConfigMaps and detects the ${trialParameters.<name>}?

Yes:

katib/pkg/ui/v1beta1/frontend/src/reducers/general.js

Lines 88 to 109 in 081db0d

	// getTrialParameters returns Trial parameters for the given YAML.
	// It parses only Trial parameters names from the YAML.
	const getTrialParameters = YAML => {
	const templateParameterRegex = '\\{trialParameters\\..+?\\}';
	let trialParameters = [];
	let trialParameterNames = [];
	let matchStr = [...YAML.matchAll(templateParameterRegex)];
	matchStr.forEach(param => {
	let newParameter = param[0].slice(param[0].indexOf('.') + 1, param[0].indexOf('}'));
	if (!trialParameterNames.includes(newParameter)) {
	trialParameterNames.push(newParameter);
	trialParameters.push({
	name: newParameter,
	reference: '',
	description: '',
	});
	}
	});
	return trialParameters;
	};

It just parses parameters for the given YAML Trial template.

[kimwnasptd]

ACK, I'll mimic this logic then.

Also, I was thinking about making the Reference field a dropdown list instead of a simple text input. My rationale was that if the field's value will be one of the hyper parameters then lets reduce the input required from the user and just list to them the available options.

My only question is if this holds for NAS Experiments. Looking at the docs I see they hint that the reference value of the trial params can be one of the hyper params, but it's not clear to me what's the case for NAS.
https://www.kubeflow.org/docs/components/katib/trial-template/#configure-trial-template-specification

Do you guys think I could make the reference input a dropdown instead or should we stick to basic text inputs?

[kimwnasptd]

Also I can confirm that I can create Experiments using the new UI's form, once it also sets the trialParameters form field.

So I'll move on to including the EarlyStopping and ResumePolicy settings, as mentioned in the beginning of the issue, which are the last missing ones for the new form to be on par with the current one.

I see that EarlyStopping

katib/pkg/apis/controller/experiments/v1beta1/experiment_types.go

Line 37 in 60f6c20

EarlyStopping *common.EarlyStoppingSpec `json:"earlyStopping,omitempty"`

accepts an arbitrary list of algorithm settings. Are these settings defined as finite list of values? Was thinking if it would make sense to pre-define these values like we do for the other algorithm settings

Values for algorithm settings

[andreyvelich]

Also, I was thinking about making the Reference field a dropdown list instead of a simple text input. My rationale was that if the field's value will be one of the hyper parameters then lets reduce the input required from the user and just list to them the available options

I think it's great improvements for the HP experiment since almost every time reference is equal to one of the parameters.
We have only one case when reference is not equal to parameters right now, it is described here.
You check the example with metadata substitution here: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/trial-metadata-substitution.yaml#L27-L48
I think we should discuss how we can provide user that functionality via submit page.

For NAS since "reference - the parameter name that experiment’s suggestion returns.", reference differs from parameters. For ENAS reference is equal to this: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/nas/enas-example-cpu.yaml#L128-L133.
For DARTS reference is equal to this: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/nas/darts-example-cpu.yaml#L45-L53.
We can setup some default parameter list for NAS experiment, I think.

WDYT @gaocegege @johnugeorge ?

[andreyvelich]

Also I can confirm that I can create Experiments using the new UI's form, once it also sets the trialParameters form field.

So I'll move on to including the EarlyStopping and ResumePolicy settings, as mentioned in the beginning of the issue, which are the last missing ones for the new form to be on par with the current one.

I see that EarlyStopping accepts an arbitrary list of algorithm settings. Are these settings defined as finite list of values? Was thinking if it would make sense to pre-define these values like we do for the other algorithm settings

Thank you for your effort @kimwnasptd, it's great.
Yes, you can find Early Stopping algorithm settings here: https://www.kubeflow.org/docs/components/katib/early-stopping/#median-stopping-rule.

[kimwnasptd]

closing this issue since #1463 has added the missing fields of the form.

The form is now on par with the current default UI.