Recommend AutoNormal guide when hessian in AutoLaplace is singular

[freddyaboulton]

Fixes #1001

To be honest, I'm not 100% sure what the gauss-newton hessian is so I'm taking my best guess from what I read here 😅

[fehiepsi]

@freddyaboulton Really sorry, I didn't find references for that feature request. I found this article is helpful and it seems that this method is called "outer product of gradients (OPG)" or BHHH estimator in literature. As I understand, this estimator assumes that the likelihood is f(latent) = (f_1(latent), ..., f_n(latent)) - then the precision matrix can be approximated by the sum/average of the outer product of gradients of those f_i. This seems to be a bit different from the formula that you use: outer product of gradients of sum(f_i). What do you think?

For now, I'm not sure what is a good way to get a covariance matrix for Laplace approximation in general cases. I guess we can defer users to AutoNormal guides in case the Hessian is singular.

[freddyaboulton]

@fehiepsi Thanks for the feedback! Do you want me to update the warning message to recommend the AutoNormal guide when the Hessian is singular or would you rather I close out the PR until we do more research?

[fehiepsi]

recommend the AutoNormal guide when the Hessian is singular

Sure, please go ahead. It is great to have a better warning message. Thanks!

[fehiepsi]

Thanks, @freddyaboulton!