Add Soft Actor-Critic (SAC)

[brahimdriss]

Description

This PR introduces SAC, following the original article, for continuous action spaces.
The current implementation was evaluated on gym Pendulum and Mujoco Hopper (v2), "solving" both environments.

Todo

• Switch to gymnasium
• Add seeding
• Handle multiple fitting
• Add benchmarks (Mujoco and classic gym control)

Original articles:
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Composable Deep Reinforcement Learning for Robotic Manipulation
Soft Actor-Critic Algorithms and Applications

Reference resources:
haarnoja/sac
haarnoja/softqlearning
openai/spinningup
vwxyzjn/cleanrl
DLR-RM/stable-baselines3

Checklist

• My code follows the style guideline
  To check :
  black --check examples rlberry *py
  flake8 --select F401,F405,D410,D411,D412 --exclude=rlberry/check_packages.py --per-file-ignores="init.py:F401",
• I have commented my code, particularly in hard-to-understand areas,
• I have made corresponding changes to the documentation,
• I have added tests that prove my fix is effective or that my feature works,
• New and existing unit tests pass locally with my changes,
• If updated the changelog if necessary,
• I have set the label "ready for review" and the checks are all green.

[brahimdriss]

Pendulum results on 10 runs. Not sure about the style of the plot, but it still can be changed later.

MuJoCo results are next.

[brahimdriss]

MuJoCo results:

Hopper-v2 : 1M timesteps - 5 runs
Training episodic return : 2247.09 ± 929.56
Walker-v2: 1M timesteps - 5 runs
Training episodic return : 4217.55 ± 417.02
HalfCheetah-v2: 1M timesteps - 5 runs
Training episodic return : 6393.02 ± 657.12

[riiswa]

Nice work, I just left I few details/suggestions that are not directly related to the SAC implementation.

[riiswa]

Just a nit, but there is some built-in Pytorch method to convert a np array to a torch tensor (torch.as_tensor, torch.from_numpy). I think torch.as_tensor allow you to specify the device directly.

[brahimdriss]

Thanks for the suggestions Waris, I tried here to be consistent with the other implementations of Rlberry, FloatTensor was used in some of them, but so was torch.from_numpy. That's why I kept it. Is there any difference in terms of performance otherwise ?

[riiswa]

Same suggestion here

[KohlerHECTOR]

It seems that something is broken with the doc in the sac branch. When trying to change the api.rst , it breaks readthedocs compilation. So I reverted. But otherwise, algorithmically speaking, SAC looks good to me.