Check Metrics Collector before mark Trial succeeded

[andreyvelich]

Here https://github.com/kubeflow/katib/blob/master/pkg/controller/v1alpha2/trial/trial_controller_util.go#L49 we mark Trial succeeded right after corresponding Trial job finished.
After that, we run GetSuggestion and inside Suggestion we, probably, have empty results, if Metrics Collector didn't collect logs.

I think, we should check that Metrics from the Trial were collected and saved in DB before mark Trial as succeeded.
Metrics collector is the part of the Trial.
What do you think @johnugeorge @richardsliu @gaocegege @hougangliu ?

[gaocegege]

Yeah, I think we should have a condition in Trial for metrics collector, for example, metrics-collected. We should check the condition before we set the trial succeeded.

[johnugeorge]

trial.status.Observation field check should be good enough to verify if metric collection is completed