[BUG] During rolling upgrade, new settings are not recognized in the cluster

[thalurur]

Describe the bug
This is not strictly a bug, but something we followed to onboard to opensearch, refer to opensearch-project/job-scheduler#20.
As stated above we deprecated old setting namespace opendistro and use plugins now in IndexManagement.

However, the issue is in a mixed cluster the new settings are not settable/updatable.
We noticed this in a 2 opendistro nodes node-1 and node-2, one OpenSearch node node-3 cluster

As part of IndexManagement plugin users can create rollup jobs. As part of rollup job run we create new rollup index if not present and set a index setting on top of the rolled up index during the first job run.

However, in mixed cluster setting if the job is run on OpenSearch node then the job fails as the os node will attempt to use the updated setting string "plugins" instead of "opendistro", this new index setting is not recognized and receiving an illegal_argument_exception. This prevents creating of new rollup indices on new nodes. The user will have to wait till the cluster is not mixed

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unknown setting [index.plugins.rollup_index] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"}],"type":"illegal_argument_exception","reason":"unknown setting [index.plugins.rollup_index] please check that any required plugins are installed, or check the breaking changes documentation for removed settings"},"status":400}

Manually tried to set any new settings (cluster/index) on a mixed cluster and seeing the same error. From observed behavior it seems that only settings allowed in a mixed cluster are old settings.

Expected behavior
New settings can be set and work during mixed cluster

Host/Environment (please complete the following information):

• OS: Linux
• Version 1.0.0

[saratvemulapalli]

Thanks for reporting this.

Settings are cluster level, if we have a mixed cluster and new/replaced settings are introduced in the new software will not work as older nodes will not understand the setting within the cluster and this is by design.
I'll add this to clarify in : https://github.com/opensearch-project/opensearch-plugins/blob/main/UPGRADING.md#backwards-compatibility

If the plugin functionality is dependent on it, i.e roll up job has to set a setting either we need to handle this in the plugin.
My thoughts:

1. Plugin should be able to handle it and set older setting during roll up, as OpenSearch 1.0.0 ISM plugin will also understand it. So it should be safe to set the older fallback setting and it should be phased out in the next future release.
2. Depending on 1., We should call this out for our customers that roll up functionality will not be supported during rolling upgrades.