Extend Config to override ini values programmatically

[acerv]

Hello, I'm trying to find a way to override custom ini values defined inside pytest.ini.

My scenario is the following: I have a custom option called --resource-enable which is implemented inside a plugin called "resources.py". If --resource-enable is used, my plugin fetch the configuration from a DB (dict type) and it has to override the pytest.ini variables defined inside the other plugins which expose the fixtures.

The system is needed in order to accomplish a complex automation scenario using Jenkins and some other proprietary tools.

In short:

• all plugins implement fixtures which are used by the tests, using custom ini values defined inside pytest.ini
• resources.py plugin fetches data from DB and override custom pytest.ini values before the tests execution

At the moment, there are no other ways but using config.addinivalue_line, but most of my ini items have the simple key=value form. A config.addinivalue would be really useful in this case.

Thanks,
Andrea

[pytestbot]

GitMate.io thinks possibly related issues are #204 (allow to dynamically modify all ini-values from plugins), #3103 (UsageError raised when specifying config override options followed by test path(s)), #2039 (Call "--collect-only" programmatically ), #2544 (Run tests truly programmatically), and #3050 (deprecate pytest.config).

[nicoddemus]

Hi @acerv,

Changing ini options on the fly feels brittle because you never know when another plugin is actually reading the value of a ini option, which might be earlier than you expect (or even change from one version to another). For example a certain option of foo plugin is read right before running the tests, so you change the value of the option in your pytest_configure() hook and things work... until foo-2.0 is released and the variable is now read during pytest_configure(), so now there's no way to know if foo will be reading the original variable or your altered variable.

Having said all that, IIUC in your use case all the parties involved are under your control, so you probably can get away with it. I suggest a couple of options:

1. Instead of holding the options using the config object, you can have an intermediate fixture which provides this options: this way your resource plugin can override them at will.

   @pytest.fixture
   def options(pytestconfig):
   
       overwritten = {}
   
       class Options:
   
           def get(self, name):
               return overwritten.get(name, pytestconfig.getini(name))
   
           def override(self, name, value):
               overwritten[name] = value
   
       return Options()        

   Now your plugins use options instead of pytestconfig:

   # previous
   @pytest.fixture
   def something(pytestconfig):
       some_value = pytestconfig.getini('some_value')
       ...
   
   # now    
   @pytest.fixture
   def something(options):
       some_value = options.get('some_value')
       ...

   The same logic follows for the resource plugin: overwrite options in the options fixture.

2. If you are willing to hack a bit, you can access the internal _inicache property of the config object:

   pytest/_pytest/config.py

   Lines 1133 to 1142 in 1b53538

   	def getini(self, name):
   	""" return configuration value from an :ref:`ini file <inifiles>`. If the
   	specified name hasn't been registered through a prior
   	:py:func:`parser.addini <_pytest.config.Parser.addini>`
   	call (usually from a plugin), a ValueError is raised. """
   	try:
   	return self._inicache[name]
   	except KeyError:
   	self._inicache[name] = val = self._getini(name)
   	return val

   But this can break between releases if you refactor how _inicache works.

I would be a little cautious about adding a new Config.setini method: as I explained before, it might be a source of confusion, and while addinivalue_line exists, it feels it was added solely to expand the list of builtin markers, which seems like it could have been done in some other way.

[acerv]

Thank you very much for your reply.

The system is still under evaluation, but to me, I don't see many options but using the _inicache, which is scaring me a bit because of side-effects on the system for a pytest update.

The options fixture is probably the best choice here and I really need to check it one more time.

For now I can say that the problem release on a particular case, which is the pytest_runtestloop implementation inside a plugin. There's a major request which requires to force particular operations just before tests are started. This is, of course, a problem when using the options fixture, because it cannot be passed through plugins via pytest_ methods implementations.

[nicoddemus]

This is, of course, a problem when using the options fixture, because it cannot be passed through plugins via pytest_ methods implementations.

Indeed this will be a problem, I'm afraid.

Perhaps @RonnyPfannschmidt can share his opinion here?

[RonnyPfannschmidt]

i would suggest to put everything that configures the values and data elements into a singe plugin object, which can be looked up by the other plugins just fine from the config

that way you have a custom object that provides the values you need exactly in the way you need without needing to mess with any of pytests config values

[acerv]

Thanks @RonnyPfannschmidt for your suggestion.

That was a plan, but I figured out that placing all configurations, which are mostly specific for other plugins, in one since configuration plugin causes major issues and a sort of "circular dependency" between plugins.

I would like to have many plugins placing their own ini configuration inside pytest, and then one single plugin overriding these configurations if (and only if) these values have been fetched from a DB by using the --resource-enable option.

This is done at the moment, by using the following code:

def addini_arg(config, name, arg):
    # override only already existing values
    if name not in config._inicache:
        return

    args = config.getini(name)
    # hack it back to the config ini-value cache
    config._inicache[name] = arg
    assert config.getini(name) != args

    sys.stdout.write("resources: updated '%s' with '%s' (previously '%s')\n"%\
        (name, args, arg))

# it's important to run this method once the ini configuration has been
# loaded: trylast=True guarantee that configuration is already loaded
# when ini configuration is overwritten with resources data
@pytest.hookimpl(trylast=True)
def pytest_configure(config):
    if not config.getoption("resource_name"):
        return

    # create resources object
    resources = _create_clara_resource(config)

    # fetch data from the resources DB
    data = resources.fetch()

    # update configuration with values from resources site
    for key in data:
        value = data[key]
        addini_arg(config, key, value)

[RonnyPfannschmidt]

as far as i understood, that plugin need to be aware of all configuration, so i should bear the responsibility for all of those as well, that way the configuration values are managed by a singular plugin you can then use as source of truth of the other plugins

with what i know so far i believe anything else will be more hackish and more fragile

[acerv]

Each ini value is well documented and exposed to the users. Then the resources plugin fetches data from a DB getting data in the json format, which is translated into a dict.

The idea is that configuring the DB correctly (with the same key name of the ini values) the plugin automatically updates these values, overriding the ones coming from pytest.ini.

[acerv]

After a deeper analysis, I guess the solution provided by @nicoddemus using the options fixture is the most elegant and it's not causing major issues or circular dependencies.

Still, I need to figure out how to handle the execution of a particular operation using fixtures inside the pytest_runtestloop.

I also thought that it's maybe possible to create a custom test, which is going to be executed before all the other tests and using a specific fixture to do the trick.

What do you think about it?

[nicoddemus]

Still, I need to figure out how to handle the execution of a particular operation using fixtures inside the pytest_runtestloop.

Can you provide more details of what you have to do there? How things work now, no fixture will have been created at this point yet: fixtures are created when tests request them.

I also thought that it's maybe possible to create a custom test, which is going to be executed before all the other tests and using a specific fixture to do the trick.

It is not clear to me what that test would do, set some global variable which would then be accessed during pytest_runtestloop?

[acerv]

Thank you @nicoddemus . After a deeper analysis and implemented the options fixture, I realized that probably the best way to approach this problem is to use an external tool, which converts DB data into a pytest.ini file. After running pytest the tests will be configured correctly, without messing up with configuration overwriting.

One limitation I found in the pytest execution, is the number of possible ini files loaded in on session.
It would be really useful in my case to have two files:

• pytest.ini: defines the built-in pytest configuration (such as addopts = -v)
• setup.cfg: defines the custom plugins configurations

In this way I can generate only setup.cfg, keeping pytest.ini unchanged. Is already possible to enable this feature in some kind of way, or I need to implement it with a plugin?

Thanks

[nicoddemus]

Hi @acerv,

Currently pytest only supports loading from a single ini file and I don't think there are hooks in place to make it load more than one file.

But if you plan to use a tool to generate a pytest.ini file, you might then consider to generate a pytest.generated.ini file instead, which contains a mix of the options from the original pytest.ini file and the options obtained from the database. This way you can run pytest passing the -c option to use pytest.generated.ini instead of pytest.ini.

[acerv]

@nicoddemus thank you very much for your support, the trick worked perfectly!
Eventually, if I want to create this file inside a pytest plugin, what's the right pytest_ method where I can do this?

[nicoddemus]

I'm afraid reading configuration from ini files happens way before hooks are called, so it's not possible under the current design.

[Zac-HD]

Closing this issue because there's a good alternative to the proposal and issue is inactive.

[tmax22]

I actually think this issue should be re-opened. in our case we are using a project that is heavily use pytest but most of the project configuration lays in config.json file. some of the config option directly related to pytest ini options and we are doing some real hacky things just to go around the problem that pytest does not provide a modern mechanism to override ini config options.

for example, if i'm writing a plugin that uses another plugin, and i want to be able to change the options passed to this plugin there is no way i cant do it with regular pytest cmd incovation. i must run python run_tests.py intermediate file that eventually would run pytest.main(run_cmd) command. but this is ugly and this way you losing the ability to run tests directly from you IDE on specific test files.