dqn vs mdqn

[KohlerHECTOR]

Description

dqn vs mdqn on mountaincar as a longtest

[KohlerHECTOR]

I have now looked at the results of comparing dqn and mdqn on mountaincar (4 fits of 1e5 steps each). And it seems mdqn is not learning.
rewards.pdf
losses.pdf
eval.pdf

[AleShi94]

I have now looked at the results of comparing dqn and mdqn on mountaincar (4 fits of 1e5 steps each). And it seems mdqn is not learning. rewards.pdf losses.pdf eval.pdf

Have you tried to run it on Acrobot? Actually, in the original paper they mostly show that M-DQN improves on DQN mostly on Atari games.

[KohlerHECTOR]

I have now looked at the results of comparing dqn and mdqn on mountaincar (4 fits of 1e5 steps each). And it seems mdqn is not learning. rewards.pdf losses.pdf eval.pdf

Have you tried to run it on Acrobot? Actually, in the original paper they mostly show that M-DQN improves on DQN mostly on Atari games.

Hello, no. It would be good to try to get a single good run on acrobot indeed. I will try it should not be too long

[KohlerHECTOR]

I have now looked at the results of comparing dqn and mdqn on mountaincar (4 fits of 1e5 steps each). And it seems mdqn is not learning. rewards.pdf losses.pdf eval.pdf

Have you tried to run it on Acrobot? Actually, in the original paper they mostly show that M-DQN improves on DQN mostly on Atari games.

Hello, no. It would be good to try to get a single good run on acrobot indeed. I will try it should not be too long

Ok it works well. I had to change the value of hyperparam "target_update_parameter" from 8000 to 0.005. 0.005 is the default value for this hyperparam in base DQN. But the default value in MDQN is now 8000. Is it normal ?
mdqn_acro_rewards.pdf
mdqn_acro_loss.pdf
mdqn_acro_eval.pdf

[KohlerHECTOR]

I have now looked at the results of comparing dqn and mdqn on mountaincar (4 fits of 1e5 steps each). And it seems mdqn is not learning. rewards.pdf losses.pdf eval.pdf

Have you tried to run it on Acrobot? Actually, in the original paper they mostly show that M-DQN improves on DQN mostly on Atari games.

Hello, no. It would be good to try to get a single good run on acrobot indeed. I will try it should not be too long

Ok it works well. I had to change the value of hyperparam "target_update_parameter" from 8000 to 0.005. 0.005 is the default value for this hyperparam in base DQN. But the default value in MDQN is now 8000. Is it normal ? mdqn_acro_rewards.pdf mdqn_acro_loss.pdf mdqn_acro_eval.pdf

Maybe we should fix default mdqn hyperparams to be the same as dqn's default ?

[AleShi94]

Maybe we should fix default mdqn hyperparams to be the same as dqn's default ?

I put the hyperparameters mentioned in their article by default. But as I already mentioned they run experiments on Atari mostly, so it could be that hyperparameters on smaller environments should be different.

[AleShi94]

Also DQN by default is using TD(lambda), while it is not implemented in M-dqn. According to what they claim in the article it is not really required with munchausen trick. But still, maybe we should compare with DQN with lambda = 0?

[KohlerHECTOR]

Also DQN by default is using TD(lambda), while it is not implemented in M-dqn. According to what they claim in the article it is not really required with munchausen trick. But still, maybe we should compare with DQN with lambda = 0?

Maybe if we want to see how good this munchausen trick is, we should launch mdqn vs dqn on acrobot using dqn with lambda = 0 indeed , and DQN default where possible ?

[TimotheeMathieu]

Also DQN by default is using TD(lambda), while it is not implemented in M-dqn. According to what they claim in the article it is not really required with munchausen trick. But still, maybe we should compare with DQN with lambda = 0?

Maybe if we want to see how good this munchausen trick is, we should launch mdqn vs dqn on acrobot using dqn with lambda = 0 indeed , and DQN default where possible ?

There are two views : either you say that the honest thing to do is to compare both algorithms while only adding the munchausen trick or you compare the best tuning of both agents. I guess it depends who you ask.

[AleShi94]

It could be an idea for another PR: compare performances of DQN for different values of lambda. We don't really know if it adds a lot considering non-zero lambdas. But for this particular test I think it is enough to do a classical DQN (lambda=0) vs Munchausen DQN. All other hyperparameters could be the same for both agents.

[KohlerHECTOR]

It could be an idea for another PR: compare performances of DQN for different values of lambda. We don't really know if it adds a lot considering non-zero lambdas. But for this particular test I think it is enough to do a classical DQN (lambda=0) vs Munchausen DQN. All other hyperparameters could be the same for both agents.

I have launched this just now.
Results:
mdqn_acro_loss.pdf

mdqn_acro_eval.pdf

mdqn_acro_rewards.pdf

I guess I will modify the long test so that it repeats this experiment on multiple seeds.
Then we should close this PR.
I guess the munchausen trick does not improve too much on acrobot, next PR we should try on minatar maybe

[AleShi94]

Agree for merging

[TimotheeMathieu]

A small comment otherwise good for me, thanks !

[mmcenta]

Ok it works well. I had to change the value of hyperparam "target_update_parameter" from 8000 to 0.005. 0.005 is the default value for this hyperparam in base DQN. But the default value in MDQN is now 8000. Is it normal ?

I think that they do the same thing in two different ways. There's two major ways of doing target network updates:

1. Copy the weights periodically, i.e. every N timesteps you copy the weights from the online NN to the target NN. In this case, it seems N = 8000.
2. Do a soft update every time such that new_weights = (1 - c) * target_weights + c * weights, in which c is a parameter functionally similar to N. In this case, c=0.005.

Also DQN by default is using TD(lambda), while it is not implemented in M-dqn. According to what they claim in the article it is not really required with munchausen trick. But still, maybe we should compare with DQN with lambda = 0?

Our DQN is a bit weird because a lot of implementations don't have a chunk_size (and consequently TD(lambda) targets). I think both are fine.

[mmcenta]

LGTM