Fast computations and HigherOrderGP

[ChristianMichelsen]

Hi!

I have a (potentially) quick question regarding fast computations when using HigherOrderGPs.

I notice the usage of fast_computations or the specific use of _fast_solves, e.g. in this issue, the tutorial and in the docstring, however, I am not sure I fully understand when and why it is used?

Is it recommended to turn in on by default (as in the issue) or is a context manager better, to avoid "over-using" it? And is it only for fitting the MLL, or what about acquisition function optimization (optimize_acqf), should it also be used for that?

fast_computations further exposes the covar_root_decomposition and log_prob settings, are they irrelevant in the context of HigherOrderGPs?

Thanks a lot in advance!

[Balandat]

So in BoTorch the _fast_solves setting is disabled by default:

botorch/botorch/__init__.py

Lines 36 to 49 in 2144440

	logger.info(
	"Turning off `fast_computations` in linear operator and increasing "
	"`max_cholesky_size` and `max_eager_kernel_size` to 4096, and "
	"`cholesky_max_tries` to 6. The approximate computations available in "
	"GPyTorch aim to speed up GP training and inference in large data "
	"regime but they are generally not robust enough to be used in a BO-loop. "
	"See gpytorch.settings & linear_operator.settings for more details."
	)
	linop_settings._fast_covar_root_decomposition._default = False
	linop_settings._fast_log_prob._default = False
	linop_settings._fast_solves._default = False
	linop_settings.cholesky_max_tries._global_value = 6
	linop_settings.max_cholesky_size._global_value = 4096
	gp_settings.max_eager_kernel_size._global_value = 4096

However, for the Kronecker algebra in HigherOrderGP to be be efficient (essentially this exploit structure in Kronecker solves / matrix decompositions), we want to locally enable this flag, which is why you see that usage. #1682 enables this inside the HigherOrderGP.posterior call so you don't have to do that there.

fast_computations further exposes the covar_root_decomposition and log_prob settings, are they irrelevant in the context of HigherOrderGPs?

They're not irrelevant; in fact the botorch default set these to False which means that both MLL and matrix root are computed using the Cholesky decomposition - which turns out can be much accelerated in the case of Kronecker structure (in conjunction with the _fast_solves setting enabled).

[ChristianMichelsen]

Thanks a lot for your answer!

So if one uses HigherOrderGP, and thus also want to exploit the Kronecker algebra, it makes sense to fit he MLL with gpt_settings.fast_computations() and likewise when one is optimizing the acquisition function?
So basically set the setting to true for all HigherOrderGP-related computations?

[Balandat]

Not exactly, it's a bit more nuanced than that, it depends on how you want to fit the model. Using approximate numerical methods will cause non-smooth objective functions, so using stochastic optimizers will be necessary - note that in the things you linked the fitting uses fit_gpytorch_mll_torch() explicitly rather than the more common fit_gpytorch_mll(). If you want to use the latter to fit the HOGP exploiting Kronecker structure you want to make sure that fast_computations setting (including covar_root_decomposition and log_prob in addition to fast_sovles) are False.

[lhch-ticra]

I'll just chip in :) - First of all, thank you for the explanations - However, what I struggle to understand is the following:
I assume that the Botorch HOGP implementation by default exploits the Kronecker structure of the overall covariance matrix using the same techniques as in the original paper:HOGP e.g. Eigenvalue Decomposition and Tucker operations to avoid ever having to form and store the full covariance matrix (As such, I assume that a cholesky factorization never becomes relevant). If this is so, why are the approximations introduced by fast_solves, covar_root_decomposition and log_prob, ever needed ? Or in other words, what part(s) of the original algorithm using eigenvalues decompositions and Tuckers operations benefit from these approximations ? If the BoTorch HOGP does not fully follow the original paper, it would also be highly beneficial to understand when this is not the case - Thank you for your time, it is highly appreciated. - Perhaps this is specifically when sampling from the posterior, e.g. when using Matherons rule ?