You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Trieste's current approach to avoiding retracing is to decorate the __call__ method of acquisition function classes with tf.function. However, this has two negative side effects:
It results in early compilation, restricting use of lazy initialisation of variables
It keeps compiled graphs in a global cache that accumulates over time (e.g. during the test runs)
To fix this we should move the use of tf.functions further out. Note that this will probably be a significant amount of work.
Following this we can also convert some variables from dynamically shaped to lazy initialised (e.g. self._eps).
The text was updated successfully, but these errors were encountered:
The graphs are stored on the instances not the class method so are ostensibly freed, though it also turns out some memory is leaked by tensorflow regardless of where the graphs are stored, meaning that runs with many compilations (such as tests) inevitably leak memory. A workaround is to split the test job into multiple runs (see Split test run into three #743 and Parallelise CI tests #749).
Given that the official tensorflow documentation includes examples like the ones in trieste, I recommend closing this issue.
Trieste's current approach to avoiding retracing is to decorate the
__call__
method of acquisition function classes withtf.function
. However, this has two negative side effects:To fix this we should move the use of tf.functions further out. Note that this will probably be a significant amount of work.
Following this we can also convert some variables from dynamically shaped to lazy initialised (e.g.
self._eps
).The text was updated successfully, but these errors were encountered: