Add hyperparameter importance chart

[chenghuzi]

Contributor License Agreement

This repository (optuna-dashboard) and Goptuna share common code.
This pull request may therefore be ported to Goptuna.
Make sure that you understand the consequences concerning licenses and check the box below if you accept the term before creating this pull request.

• I agree this patch may be ported to Goptuna by other Goptuna contributors.

Reference Issues/PRs

Fixs #25

Add hyperparameter importance chart

There're two ways to add hyperparameter importance chart. One is to implement the evaluator which completely mirrors the one used by optuna via typescript and another is to use the optuna optuna.importance.get_param_importances API directly.

The former makes the logic cleaner but involves extra computation in the browser, and implementing a complete parameter importance function by ts requires an update whenever the optuna API changes.

The latter one is much more flexible and only requires introducing another API call. But the con is it takes longer for the server to get the importance result.

Here we take the second approach and only consider the single objective function case. For future multi-objective case, we can extend the API with query parameters representing params and target selected by the user.

[c-bata]

Wow, great! I'll review tomorrow.

There're two ways to add hyperparameter importance chart. One is to implement the evaluator which completely mirrors the one used by optuna via typescript and another is to use the optuna optuna.importance.get_param_importances API directly.

The former makes the logic cleaner but involves extra computation in the browser, and implementing a complete parameter importance function by ts requires an update whenever the optuna API changes.

The latter one is much more flexible and only requires introducing another API call. But the con is it takes longer for the server to get the importance result.

Here we take the second approach and only consider the single objective function case. For future multi-objective case, we can extend the API with query parameters representing params and target selected by the user.

I completely agree with your decision. It is quite difficult to port an importance evaluator into TypeScript because fANOVA evaluator requires the random forest regression model.

[c-bata]

Overall looks good and easy to read! I left some minor suggestions.

[c-bata]

param-importance.mp4

Please set deps to avoid sending API requests at every render.

Suggested change

	})
	}, [studyId])

[chenghuzi]

Oh thanks! I missed that part, re-rendering could be unnecessary.

[chenghuzi]

I was just thinking sometimes studyId could not be a proper dependency as trials may get updated while the studyId remains the same while parameter importance requires a re-evaluation.

[c-bata]

LGTM!! For further improvements, we can work on following tasks in another pull requests:

• In-memory cache: get_param_importances() is computationally expensive because of the training of RandomForest model. So it's better to cache the result at server-side like trials are cached.
• Multi-objective support: It seems that Optuna's importance module does not still support multi-objective study. So we need to send a pull request to Optuna for this. It seems to be supported.
• Calculate hyperparameter importances on Web Assembly: After I implement RandomForest and fANOVA algorithm in Goptuna, we can calculate hyperparameter importances at WebAssembly. It's a bit technically difficult, but interesting.