Trainer.test() does not allow to save logs

[Varal7]

🐛 Bug

When we use trainer.test() on a fresh trainer, the call to self.log_metrics(log_metrics, {}, step) fails to produce logs.

To Reproduce

Steps to reproduce the behavior:

Code sample

model = ...  
test_dl = ... 
logger = ...
trainer = pl.Trainer(
   logger=logger
)
trainer.test(model, test_dataloaders=test_dl)

Expected behavior

Metrics are logged.

[Varal7]

Because of the way aggregate_metrics is implemented, logs are only produced when we are leaving the current step. Because there is only one step in testing, the logs are stored in self._metrics_to_agg = [metrics] but never logged.

[edenlightning]

@ justusschock any ideas?

[Varal7]

Btw, I was able to log my logs by adding self.logger.log_metrics(logs) at the end of test_epoch_end but I believe this should be the default.

[edenlightning]

@Varal7 would you be able to send a PR with the fix?

[williamFalcon]

results = trainer.test()

results is a dict with the results

[wconnell]

if you encounter no logging on end of epoch it may be because you are using ddp, requiring explicit usage of test_step_end see the documentation here:

def test_step(self, batch, batch_idx):
    X, y = batch
    y_hat = self(X)
    return y_hat, y

def test_step_end(self, output_results):
    y_hat, y  = output_results
    loss = criterion(y_hat, y)
    self.log(loss, epoch_end=True)