[BUG] AttributeError: 'ExperimentWriter' object has no attribute 'add_figure'

[bgrey001]

• PyTorch-Forecasting version: 0.10.3
• PyTorch version: 1.13.0
• PyTorch Lightnight version: 1.9.0 (downgraded due to issues posted here 'tuple' object has no attribute 'items' in models #1255)
• Python version: 3.10.8
• Operating System: macOS BigSur and Ubuntu

Stuck on my first few tutorials of PyTorch Forecasting because of a few bugs. I've sidestepped a KeyError: 'radam_buffer' error by introducing an optimizer into the model building from_dataset(). Would greatly appreciate some help!

Expected behavior

When calling fit on the model (using the exact code from the DeepAR tutorial), expected to fit the model and see a model summary according to the tutorial.

Actual behavior

However, error encountered after a single plot is produced and the following:

self.log_prediction(

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_forecasting/models/base_model.py", line 731, in log_prediction
    self.logger.experiment.add_figure(

AttributeError: 'ExperimentWriter' object has no attribute 'add_figure'

Code to reproduce the problem

net = DeepAR.from_dataset(
    training,
    learning_rate=0.1,
    optimizer='Adam',
    log_interval=10,
    log_val_interval=1,
    hidden_size=30,
    rnn_layers=2,
    loss=MultivariateNormalDistributionLoss(rank=30),
)

# =============================================================================
# fit the model
# =============================================================================
trainer.fit(
    net,
    train_dataloaders=train_dataloader,
    val_dataloaders=val_dataloader,
)

Paste the command(s) you ran and the output. Including a link to a colab notebook will speed up issue resolution.
If there was a crash, please include the traceback here.
The code used to initialize the TimeSeriesDataSet and model should be also included.

TRACEBACK:

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "/Users/BenedictGrey/Documents/GitHub/algo_trading/models/python/pytorch_forecasting/DeepAR.py", line 160, in <module>
    trainer.fit(

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run_train
    self._run_sanity_check()

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1267, in _run_sanity_check
    val_loop.run()

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
    output = fn(*args, **kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_lightning/strategies/strategy.py", line 390, in validation_step
    return self.model.validation_step(*args, **kwargs)

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_forecasting/models/base_model.py", line 420, in validation_step
    log.update(self.create_log(x, y, out, batch_idx))

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_forecasting/models/deepar/__init__.py", line 344, in create_log
    log = super().create_log(

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_forecasting/models/base_model.py", line 469, in create_log
    self.log_prediction(

  File "/Users/BenedictGrey/opt/miniconda3/envs/py310-env/lib/python3.10/site-packages/pytorch_forecasting/models/base_model.py", line 731, in log_prediction
    self.logger.experiment.add_figure(

AttributeError: 'ExperimentWriter' object has no attribute 'add_figure'

CODE TO REPRODUCE:

import warnings
warnings.filterwarnings("ignore")

import matplotlib.pyplot as plt
import pandas as pd
import pytorch_lightning as pl
from pytorch_lightning.callbacks import EarlyStopping
import torch

from pytorch_forecasting import Baseline, DeepAR, TimeSeriesDataSet
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_forecasting.data.examples import generate_ar_data
from pytorch_forecasting.metrics import SMAPE, MultivariateNormalDistributionLoss

# =============================================================================
# load data
# =============================================================================
data = generate_ar_data(seasonality=10.0, timesteps=400, n_series=1, seed=42)
data["static"] = 2
data["date"] = pd.Timestamp("2020-01-01") + pd.to_timedelta(data.time_idx, "D")
data.head()

data = data.astype(dict(series=str))


# create dataset and dataloaders
max_encoder_length = 60 # assuming this is AR term so lag?
max_prediction_length = 20 # n_steps for forecasting or prediction?

training_cutoff = data["time_idx"].max() - max_prediction_length # this seems to be finding latest date and then removing the max_pred length to find the size of the training set

context_length = max_encoder_length
prediction_length = max_prediction_length

training = TimeSeriesDataSet(
    data[lambda x: x.time_idx <= training_cutoff],
    time_idx="time_idx",
    target="value",
    categorical_encoders={"series": NaNLabelEncoder().fit(data.series)},
    group_ids=["series"],
    static_categoricals=[
        "series"
    ],  # as we plan to forecast correlations, it is important to use series characteristics (e.g. a series identifier)
    time_varying_unknown_reals=["value"],
    max_encoder_length=context_length,
    max_prediction_length=prediction_length,
)


validation = TimeSeriesDataSet.from_dataset(training, data, min_prediction_idx=training_cutoff + 1)
batch_size = 128

# synchronize samples in each batch over time - only necessary for DeepVAR, not for DeepAR
train_dataloader = training.to_dataloader(
    train=True, batch_size=batch_size, num_workers=0, batch_sampler="synchronized"
)
val_dataloader = validation.to_dataloader(
    train=False, batch_size=batch_size, num_workers=0, batch_sampler="synchronized"
)


# =============================================================================
# calculate baseline error
# =============================================================================
# calculate baseline absolute error
actuals = torch.cat([y[0] for x, y in iter(val_dataloader)])
baseline_predictions = Baseline().predict(val_dataloader)
SMAPE()(baseline_predictions, actuals)


# =============================================================================
# init network
# =============================================================================
pl.seed_everything(42)
import pytorch_forecasting as ptf

trainer = pl.Trainer(gpus=0, gradient_clip_val=1e-1)
net = DeepAR.from_dataset(
    training, learning_rate=3e-2, optimizer='Adam', hidden_size=30, rnn_layers=2, loss=MultivariateNormalDistributionLoss(rank=30)
)



# =============================================================================
# find optimal lr
# =============================================================================
res = trainer.tuner.lr_find(
    net,
    train_dataloaders=train_dataloader,
    val_dataloaders=val_dataloader,
    min_lr=1e-5,
    max_lr=1e0,
    early_stop_threshold=100,
)
print(f"suggested learning rate: {res.suggestion()}")
fig = res.plot(show=True, suggest=True)
fig.show()
net.hparams.learning_rate = res.suggestion()

# =============================================================================
# early stopping
# =============================================================================
early_stop_callback = EarlyStopping(monitor="val_loss", min_delta=1e-4, patience=10, verbose=False, mode="min")
trainer = pl.Trainer(
    max_epochs=30,
    gpus=0,
    enable_model_summary=True,
    gradient_clip_val=0.1,
    callbacks=[early_stop_callback],
    limit_train_batches=50,
    enable_checkpointing=True,
)

# =============================================================================
# new net with opt learning rate
# =============================================================================
net = DeepAR.from_dataset(
    training,
    learning_rate=0.1,
    optimizer='Adam',
    log_interval=10,
    log_val_interval=1,
    hidden_size=30,
    rnn_layers=2,
    loss=MultivariateNormalDistributionLoss(rank=30),
)

# =============================================================================
# fit the model
# =============================================================================
trainer.fit(
    net,
    train_dataloaders=train_dataloader,
    val_dataloaders=val_dataloader,
)

[deividasovs]

Had the same issue, somehow fixed by changing these package versions:

Jupyter VsCode Extension: v2022.11.1003412109 
PyTorch Lightning: v1.9.2
numpy: 1.22.0 

[yingweiy]

Unfortunately when I set the PyTorch Lightning to 1.9.2, it complains "AttributeError: 'tuple' object has no attribute 'items'". According to the fix to this issue, the Lightning should be 1.9.0 to avoid this "tuple" problem. Now the problem becomes circular, and no solution yet. :(

[Marc-schaefer00]

Hi, is there still no solution to the circular problem?

[ashnair1]

This could be because you are using a different logger. add_figure is a function of TensorBoardLogger

[xhxlab]

Hi, is there still no solution to the circular problem?

[st-nickkimer]

@xhxlab This workaround worked for me. I pip installed tensorboard and created my own initialized tensorboardlogger and specified it in the trainer object.

from pytorch_lightning import loggers as pl_loggers
tensorboard = pl_loggers.TensorBoardLogger('./')

trainer = pl.Trainer(
    max_epochs=30,
    enable_model_summary=True,
    gradient_clip_val=0.1,
    callbacks=[early_stop_callback],
    limit_train_batches=50,
    enable_checkpointing=True,
    logger=tensorboard
)

[ManuelRoeder]

I was able to fix the same error by installing tensorboard via pip. PL seems to check for the tensorboard installation.

[STLBOX]

i solve this problem by following steps：
0. pip install tensorboard. installing the pakage of tensorboard in your env is must !!!

in base_model.py, to import SummaryWriter. just like:
from torch.utils.tensorboard.writer import SummaryWriter
2.because SummaryWriter.add_figure() is the correct fun.you should replace the function of the original one: self.logger.experiment.add_figure(), just like:
3.create the obj in log_prediction(): add_fig = SummaryWriter()
4.replace original code in log_prediction():
add_fig.add_figure( # self.environment.add_figure
f"{self.target_names[idx]} {tag}",
f,
global_step=self.global_step,
)
remember debug use pycharm or other ide to get in the base_model.py.

[ewth]

I realise this thread is a little dated, but I've just come across the same issue. I'm going to submit a bugfix PR.

[fkiraly]

Thanks, @ewth!

I presume the bug is still there on 1.1.0 then?

[ewth]

It is indeed @fkiraly.

I went to create a bug fix PR but having some issues with poetry and pre-commit. Working to get my local env setup and will issue the PR.

[fkiraly]

@ewth, could you kindly open a pull request? Then I might also advise with the pre-commit, we have removed poetry from the dev tools.

[ewth]

@fkiraly done :)

There's a similar error for add_embedding:

Traceback (most recent call last):
  File ".../test_tft.py", line 155, in <module>
    trainer.fit(
  File ".../lightning/pytorch/trainer/trainer.py", line 538, in fit
    call._call_and_handle_interrupt(
  File ".../lightning/pytorch/trainer/call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lightning/pytorch/trainer/trainer.py", line 574, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File ".../lightning/pytorch/trainer/trainer.py", line 991, in _run
    call._call_lightning_module_hook(self, "on_fit_end")
  File ".../lightning/pytorch/trainer/call.py", line 167, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File ".../pytorch_forecasting/models/temporal_fusion_transformer/__init__.py", line 517, in on_fit_end
    self.log_embeddings()
  File ".../pytorch_forecasting/models/temporal_fusion_transformer/__init__.py", line 866, in log_embeddings
    self.logger.experiment.add_embedding(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ExperimentWriter' object has no attribute 'add_embedding'

I've addressed it in the same PR, but can split it off into its own issue if you would prefer.

[ewth]

For additional context, this bug occurs if tensorboard is not installed. This is possibly a downstream effect of lightning removing tensorboardX as a dependency:

Starting from v1.9.0, `tensorboardX` has been removed as a dependency of the `lightning.pytorch` package, due to potential conflicts with other packages in the ML ecosystem. For this reason, `logger=True` will use `CSVLogger` as the default logger, unless the `tensorboard` or `tensorboardX` packages are found. Please `pip install lightning[extra]` or one of them to enable TensorBoard support by default

To replicate:

1. Ensure tensorboard is not installed.
2. Create TimeSeriesDataSet datasets.
3. Create a TemporalFusionTransformer model.
4. Run Trainer.fit() on the model with the dataset dataloaders.

[ewth]

@fkiraly my apologies, there are actually some differences between this and what I've attempted to fix. The error is very similar to what I'm experiencing and so I jumped on it, but for my situation:

• Python 3.12.7
• PyTorch 2.4.1
• Lightning 2.4.0
• PyTorch Forecasting 1.1.1
• TemporalFusionTransformer model.
• It occurs when self.logger.experiment.add_figure is called.

Should I create a new bug report and remove the current PR?

[fkiraly]

@ewth, good question. I think we should keep the current PR as it can deal with both issues, but abstract the check into a _logger_supports method. That way, we have a single entry point to add changes in logic on how we access the logger, as the logger interfaces seem to be highly unstable across version combinations of the key dependencies.

I left some comments there, on #1694