RoboschoolHumanoidFlagrun-v1 flies in an endless summersault?

[leedtan]

I believe the RoboschoolHumanoidFlagrun-v1 environment has a bug in the physics engine.

I'm trying to train humanoid RoboschoolHumanoidFlagrun-v1 via actor critic and sometimes visualize what he's learning. I have the "done" flag disabled so that he can run for a few steps even after the episode is over, to view how the environment works in weird situations.

Sometimes I see him do flips/summersaults/fly into the air and away from the screen very quickly.

It seems hard for him to do because he doesn't do it often, but I believe there is a bug in the physics engine that allows him to fly and due to this bug, my agent ignores the real goal and tries to learn to fly because that is a quick and easy way to move quickly toward the flag.

Is this a known bug? Are there specific states I could track when I try to demonstrate this occurring? This seems like a classic bug according to the saying "if you want to find bugs in your physics engine, train an RL agent on the environment and see if it starts to fly".