[BUG] Warning message is logged when multiple OpenSearch nodes are running in a single JVM

[lukasz-soszynski-eliatra]

Describe the bug
Warn message ApplierService#updateTask][T#1] WARN ClusterApplierService:628 - failed to notify ClusterStateListener is logged when more than one node of OpenSearch works in a single JVM

To Reproduce
The issue is related to the integration test framework which is created to test the OpenSearch security plugin. To reproduce the problem:

1. Checkout code which contains a integration test framework: New test framework security#1967
2. Run a test which starts more than one node of OpenSearch e.g. org.opensearch.security.SecurityRolesTests

Expected behavior
The warning message is not present in the logs

Host/Environment (please complete the following information):

Additional context
The problem is related to Log4j2 plugin org.opensearch.common.logging.NodeAndClusterIdConverter and its static field nodeAndClusterId. During the OpenSearch start time, each node set its node and cluster id in the field nodeAndClusterId. Unfortunately, this operation can be performed only once because the field type nodeAndClusterId is SetOnce. Therefore, if more than one node works in the same JVM then the warn message is printed.

Please see also:

• New test framework security#1967 (comment)
• It is possible to disable NodeAndClusterIdConverter plugin. #4322