Multi-fidelity BO tutorial doesn't work?

[samuelkim16]

I am going off of the multi-fidelity BO tutorial, which compares multi-fidelity knowledge gradient (KG) with the traditional expected improvement (EI) on just the highest fidelity. At the end, I plot the optimization curves as a function of cost (in the case of multi-fidelity BO, only tracking the highest-fidelity evaluations for the best value so far), but it seems that single-fidelity EI outperforms multi-fidelity KG. Is this not the right way to track this?

fixed_cost = 5.0

cost_ei = []
best_y_ei = []

cumulative_cost = 0

for xi, yi in zip(train_x_ei, train_obj_ei):
    yi = yi[0].item()
    costi = xi[-1].item()
    cumulative_cost += costi + fixed_cost

    if costi == 1:
        cost_ei.append(cumulative_cost)

        if not best_y_ei or yi > best_y_ei[-1]:
            best_y_ei.append(yi)
        else:
            best_y_ei.append(best_y_ei[-1])



cost_mf = []
best_y_mf = []

cumulative_cost = 0

for xi, yi in zip(train_x, train_obj):
    yi = yi[0].item()
    costi = xi[-1].item()
    cumulative_cost += costi + fixed_cost

    if costi == 1:
        cost_mf.append(cumulative_cost)

        if not best_y_mf or yi > best_y_mf[-1]:
            best_y_mf.append(yi)
        else:
            best_y_mf.append(best_y_mf[-1])

import matplotlib.pyplot as plt

plt.figure()
plt.plot(cost_ei, best_y_ei, marker='o', label="Single-fidelity, EI")
plt.plot(cost_mf, best_y_mf, marker='o', label="Multi-fidelity, KG")
plt.xlabel("Cost")
plt.ylabel("Objective")
plt.legend()

Edit:

I just looked at the output of an earlier cell in the notebook where it makes a final recommendation from the multi-fidelity KG, and the output is the following:

recommended point:
tensor([[0.315, 0.138, 0.348, 0.273, 0.300, 0.617, 1.000]],
       dtype=torch.float64)

objective value:
tensor([3.048], dtype=torch.float64)

total cost: 136.0

The objective value is better than it had found during optimization or what EI was able to find. So somehow the GP "knows" about the best objective, but never actually evaluates it? Am I interpreting this correctly?

[Balandat]

The objective value is better than it had found during optimization or what EI was able to find. So somehow the GP "knows" about the best objective, but never actually evaluates it? Am I interpreting this correctly?

Yes, this is correct. MFKG avoids evaluating the high fidelities due to the high cost. So your analysis about how well it does needs to be based on the points it would recommend at each step (not what it evaluates as part of the optimization). Take a look at the ["Make a final recommendation¶"](https://botorch.org/tutorials/discrete_multi_fidelity_bo#Make-a-final-recommendation) section of the tutorial. You'll want to call get_recommendation() in each step based on the model fit on the data observed up to that point. You can compare performance of the two approaches by comparing that recommendation with the value chosen by EI.

This is often referred to as considering "inference regret" rather than "simple regret" - the former is based on what the model believes is best, the latter is simply based on the points that have been evaluated so far.

[Balandat]

Hmm so looking at that tutorial, it seems that the cost is 5.0 + s, where s \in {0.5, 0.75, 1.0}. This means that the cost difference between low and high fidelity observations is tiny and only ranges between 5.5 and 6.0. I haven't looked into this in more detail, but my hunch is that this is simply not a particularly meaningful problem for illustrating the benefits of multi-fidelity optimization (which begs the question why we have this particular setup as the example - I'll look into that). I'd recommend reducing the fixed cost to something much less (e.g. 0.25 or something like that) and trying this again (for a smaller total cost budget) and see how that changes the results - I'd be quite interested in seeing that.

[Balandat]

cc @SebastianAment, @ltiao, @dme65

[samuelkim16]

Thank you for the recommendation, I realized I forgot to follow up on this discussion. I set the fixed cost to 0.25, and it seems to improve the performance for multi-fidelity optimization. Interestingly, the single-fidelity version of knowledge gradient (SF-KG) achieves the lowest regret in the end. Benchmarking against other problems, I've also found the performance of multi-fidelity optimization in general to depend highly on the problem - in many cases not helping or sometimes performing extremely poorly.

Below are results for the best objective and simple regret as a function of cost for the augmented Hartmann problem (using the same setup as in the tutorial). For KG, the best objective is calculated by the simple regret in each iteration (as opposed to the inference regret). The shaded areas represent +/- 1 standard deviation, over 10 trials.

[Balandat]

Thanks for the follow-up on this.

Interestingly, the single-fidelity version of knowledge gradient (SF-KG) achieves the lowest regret in the end.

The CIs here are still quite wide so this could just be due to randomness in the evaluation. I guess also the fact that this is in the very high cost regime makes this less practically relevant (typically we don't have a ton of budget which is why one would use MF in the first place).

I've also found the performance of multi-fidelity optimization in general to depend highly on the problem - in many cases not helping or sometimes performing extremely poorly.

Yeah, unfortunately this is a general rule and not exclusive to MF methods.

[Balandat]

Adjusted the cost function in the tutorial based on this discussion: #2711