Prevented accidental alteration of env_spec_kwargs in make_vec

[TimSchneider42]

Description

In make_vec, env_spec_kwargs is first stripped of all arguments which do not belong to the underlying environment (num_envs, vectorization_mode, ...). Then create_single_env is defined, where env_spec_kwargs is used to create the environment, which is then passed to either SyncVectorEnv or AsyncVectorEnv. After those vector environments are created, the stripped arguments are added to env_spec_kwargs again.

AsyncVectorEnv keeps a reference to create_single_env in a publicly accessible field AsyncVectorEnv.env_fns. The problem is now that because the stripped arguments were added to env_spec_kwargs again, create_single_env does not work anymore, as it will now try to pass arguments such as num_envs to the environment constructor.

In my workflow, I need to access a single environment of AsyncVectorEnv, e.g.:

import gymnasium
env = gymnasium.make_vec("CartPole-v1", vectorization_mode=gymnasium.VectorizeMode.ASYNC, num_envs=4)
env.env_fns[0]()

produces

Traceback (most recent call last):
  File ".../gymnasium/envs/registration.py", line 741, in make
    env = env_creator(**env_spec_kwargs)
TypeError: CartPoleEnv.__init__() got an unexpected keyword argument 'num_envs'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File ".../example.py", line 3
    env.env_fns[0]()
  File ".../gymnasium/envs/registration.py", line 902, in create_single_env
    single_env = make(env_spec, **env_spec_kwargs.copy())
  File ".../gymnasium/envs/registration.py", line 753, in make
    raise type(e)(
TypeError: CartPoleEnv.__init__() got an unexpected keyword argument 'num_envs' was raised from the environment creator for CartPole-v1 with kwargs ({'num_envs': 4, 'vectorization_mode': 'async'})

The fix is to copy the env_spec_kwargs dictionary before adding the stripped arguments again.

Type of change

• Bug fix (non-breaking change which fixes an issue)

[pseudo-rnd-thoughts]

Thanks @TimSchneider42 for the PR, the change makes sense.
My only question is if this should be a deepcopy? or just a copy? As the kwargs could be objects, etc

[TimSchneider42]

I think a deepcopy would not be a good idea here, as it would mean that objects would be passed by reference during the initial call of create_single_env inside make_vec but not anymore in subsequent calls to env.env_fns[0](). To me as a user, I would not expect arguments I pass to make_vec to be copied at all. So I think a flat copy is the best choice here.