[retiarii] support visualize model space with the hpo chart on webui #4304

QuanluZhang · 2021-11-10T12:02:00Z

Description

support the hyper-parameter chart for retiarii experiment.

Checklist

search space generated by retiarii engine is stored to db through nnimanager
add a sub dict in trial configuration for visualization
modify webui for correct visualization of the hyper parameter chart
test case
doc: no doc should be updated

How to test

run the multi-trial nas example and view the hyper-parameter chart

fix hyper-parameter graph issue in retiarii experiment

liuzhe-lz · 2021-11-18T06:35:28Z

nni/runtime/protocol.py

@@ -27,6 +27,7 @@ class CommandType(Enum):
    SendTrialJobParameter = b'SP'
    NoMoreTrialJobs = b'NO'
    KillTrialJob = b'KI'
+    ReportSearchSpace = b'RS'


I thought the visual search space could be directly embedded into original search space, rather than requiring a new channel.

Because we should add a new transmission direction, i.e., sending search space from tuner/advisor to nnimanager. Previously, search space is sent from nnimanager to tuner/advisor

We already have update search space in REST API. See Experiment.update_search_space() for an example implementation.
I think Retiarii should report search space use this method.

update search space is from nnimanager to advisor, not advisor to nnimanager

Experiment.update_search_space() is from experiment launcher (RetiariiExperiment) to NNI manager, and then to dispatcher.

updated to using this function

ultmaster · 2021-11-18T09:41:30Z

nni/retiarii/execution/python.py



 class PythonGraphData:
    def __init__(self, class_name: str, init_parameters: Dict[str, Any],
-                 mutation: Dict[str, Any], evaluator: Evaluator) -> None:
+                 mutation: Dict[str, Any], evaluator: Evaluator, visual_hp: dict = None) -> None:


visual_hp can be inferred from mutation I think.

ultmaster · 2021-11-18T09:43:04Z

nni/retiarii/execution/python.py


    def dump(self) -> dict:
        return {
            'class_name': self.class_name,
            'init_parameters': self.init_parameters,
            'mutation': self.mutation,
-            'evaluator': self.evaluator
+            'evaluator': self.evaluator,
+            '_visual_hyper_params_': self.visual_hp


I don't think visual_hyper_params is so special that it needs to be quoted with underline.

I think that depends on whether _visual_hyper_params_ is designed to be Retiarii only.
In HPO scenario all other stuffs are meant for tuning algorithm, but this field is for web UI.

some other stuffs are meant for trial. in hpo, those stuffs are visualizable, but in retiarii they are not. that is why we need a new field. "visual_hyper_params" is special, because it delivers redundant data

ultmaster · 2021-11-18T09:43:56Z

nni/retiarii/execution/base.py


 class BaseGraphData:
-    def __init__(self, model_script: str, evaluator: Evaluator) -> None:
+    def __init__(self, model_script: str, evaluator: Evaluator, visual_hp: dict = None) -> None:


Suggest renaming it into something like mutation_summary. It's not for visualization purpose only.

ultmaster · 2021-11-18T09:44:24Z

nni/retiarii/integration_api.py

+    """
+    Execution engine uses this API to report the search space extracted by
+    strategy to nnimanager which stores the search space into db,
+    so that webui can access the search space like HPO


Missing period.

ultmaster · 2021-11-18T09:48:09Z

nni/retiarii/strategy/base.py

+            if mutator.label is not None:
+                label = mutator.label
+            else:
+                label = f'auto_label_{auto_label_seq}'


Where does auto_label come from? I think this logic is already done in dry_run_for_search_space, and self is never used in this function. Suggest refact it into a util.

moved to util. dry_run_for_search_space cannot be used as the generated search space should follow our search space format

cruiseliu · 2021-12-05T07:02:56Z

nni/retiarii/execution/base.py

        self.model_script = model_script
        self.evaluator = evaluator
+        self.mutation_summary = mutation_summary


This introduces a new term so it needs doc/comment to describe what is "mutation summary".

matluster · 2021-12-05T08:06:10Z

nni/retiarii/strategy/base.py



 class BaseStrategy(abc.ABC):

+    @classmethod
+    def get_model_space(self, base_model: Model, applied_mutators: List[Mutator]) -> None:


This looks really weird. BaseStrategy is an interface, yet you put concrete implementations into it.

matluster · 2021-12-05T08:06:54Z

nni/retiarii/experiment/pytorch.py

@@ -193,6 +193,8 @@ def _start_strategy(self):
        )

        _logger.info('Start strategy...')
+        search_space = BaseStrategy.get_model_space(base_model_ir, self.applied_mutators)


Should call dry_run_for_formated_search_space here. Not sure whether every search space can be successfully formatted though.

ultmaster · 2021-12-07T03:35:30Z

ts/webui/src/static/model/trial.ts

@@ -27,13 +28,14 @@ import {
 * @returns Parsed structured parameters and unexpected entries
 */
 function inferTrialParameters(
-    paramObj: object,
+    paramObj: RetiariiParameter,


Why is this RetiariiParameter? How about normal trials?

I think the changes above are not needed. In L35, we can put:

'mutation_summary' in paramObj ? (paramObj as any).mutation_summary : paramObj;

@ultmaster @QuanluZhang also fix this in PR #4314

ultmaster · 2021-12-08T08:36:39Z

nni/retiarii/execution/base.py

 class BaseGraphData:
-    def __init__(self, model_script: str, evaluator: Evaluator) -> None:
+    """
+    Parameters


Attributes~~Parameters~~

QuanluZhang added 6 commits November 10, 2021 20:00

first-round update

69ec88c

retiarii search space are stored in db through nnimanager

a317fd0

add comments

c0bc292

update the logic of search space generation

1516035

update

4150951

add a sub dict for visualizing trial configuration

486c6ce

QuanluZhang marked this pull request as ready for review November 11, 2021 11:08

liuzhe-lz requested review from ultmaster and liuzhe-lz November 12, 2021 08:34

liuzhe-lz mentioned this pull request Nov 12, 2021

NNI 2021 Oct~Nov Iteration Plan #4211

Closed

86 tasks

Lijiaoa and others added 3 commits November 12, 2021 09:59

update

2bd2e7f

Merge branch 'master' of github.com:microsoft/nni into dev-nas-config

31ec053

Merge pull request #8 from Lijiaoa/dev-retiarii-hyper

29343b3

fix hyper-parameter graph issue in retiarii experiment

Lijiaoa mentioned this pull request Nov 15, 2021

[webui] Improve trial parameter for retiarii experiment #4314

Merged

10 tasks

liuzhe-lz reviewed Nov 18, 2021

View reviewed changes

ultmaster reviewed Nov 18, 2021

View reviewed changes

QuanluZhang added 6 commits December 3, 2021 21:23

resolve comments

bed2f2c

update webui accordingly

9414c2e

fix minor

a7dcf48

update

34c1bd9

minor

380a38e

fix pylint

6c2e8f6

cruiseliu reviewed Dec 5, 2021

View reviewed changes

matluster reviewed Dec 5, 2021

View reviewed changes

QuanluZhang added 3 commits December 6, 2021 17:04

fix pylint in cgo

75cc679

fix ut

23352cd

fix import

aae0206

ultmaster reviewed Dec 7, 2021

View reviewed changes

QuanluZhang added 2 commits December 7, 2021 12:31

Merge branch 'master' of github.com:microsoft/nni into dev-nas-config

f9ef69d

resolve comments

fd4fdc4

QuanluZhang added 4 commits December 8, 2021 08:38

fix webui

5e61502

minor

2cf99cd

fix eslint

45b7caf

fix eslint

0619c7d

ultmaster reviewed Dec 8, 2021

View reviewed changes

ultmaster approved these changes Dec 8, 2021

View reviewed changes

minor

cd6cfaa

ultmaster closed this Dec 10, 2021

ultmaster reopened this Dec 10, 2021

liuzhe-lz approved these changes Dec 10, 2021

View reviewed changes

liuzhe-lz merged commit 357ec6e into microsoft:master Dec 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[retiarii] support visualize model space with the hpo chart on webui #4304

[retiarii] support visualize model space with the hpo chart on webui #4304

QuanluZhang commented Nov 10, 2021 •

edited

Loading

liuzhe-lz Nov 18, 2021

QuanluZhang Nov 18, 2021

liuzhe-lz Nov 24, 2021

QuanluZhang Dec 3, 2021

liuzhe-lz Dec 3, 2021

QuanluZhang Dec 3, 2021

ultmaster Nov 18, 2021

ultmaster Nov 18, 2021

liuzhe-lz Nov 24, 2021

QuanluZhang Dec 3, 2021

ultmaster Nov 18, 2021

QuanluZhang Dec 3, 2021

ultmaster Nov 18, 2021

ultmaster Nov 18, 2021

QuanluZhang Dec 3, 2021

cruiseliu Dec 5, 2021 •

edited

Loading

QuanluZhang Dec 8, 2021

matluster Dec 5, 2021

matluster Dec 5, 2021

ultmaster Dec 7, 2021

QuanluZhang Dec 7, 2021

ultmaster Dec 7, 2021

Lijiaoa Dec 8, 2021

ultmaster Dec 8, 2021

[retiarii] support visualize model space with the hpo chart on webui #4304

[retiarii] support visualize model space with the hpo chart on webui #4304

Conversation

QuanluZhang commented Nov 10, 2021 • edited Loading

Description

Checklist

How to test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cruiseliu Dec 5, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuanluZhang commented Nov 10, 2021 •

edited

Loading

cruiseliu Dec 5, 2021 •

edited

Loading