[BUG] Security plugin APIs error out within the BWC test suite with 400 no handler found for uri

[parasjain1]

What is the bug?
When working on #2802 I'm running into the following issue while creating a test role within BWC suite -

2> org.opensearch.client.ResponseException: method [PUT], host [http://127.0.0.1:34017/], URI [_plugins/_security/api/roles/test-dls-fls-role], status line [HTTP/1.1 400 Bad Request]
{"error":"no handler found for uri [_opendistro/_security/api/roles/test-dls-fls-role] and method [PUT]"}
The role is being created as part of a separate test in the same suite. Further, it looks like none of the security plugin APIs are accessible and error out with no handler found for uri. Bulk and Search APIs are working.

How am I running the test suite ?

current=3.0.0.0
previous=2.10.0.0
./gradlew bwcTestSuite -Dtests.security.manager=false -Dbwc.version.previous=$previous -Dbwc.version.next=$current -i
Log file -
security-bwc (1).log

Here's the code on my working branch. It's in an intermediate state with pending changes -
https://github.com/parasjain1/opensearch-security/blob/parasjaz-custom-serialization-bwc/bwc-test/src/test/java/org/opensearch/security/bwc/SecurityBackwardsCompatibilityIT.java#L97

Need help to understand what could be causing it? Have also tried the _opendistro prefix and without the prefix _security/_api/, which also results in the same error.

[parasjain1]

PR to let BWC tests run in Github actions -
#3054

[peternied]

When BWC tests fail they put log output into the console, I'm seeing the following line that seems troublesome, this would imply that the security plugin features won't be available, doing more digging

» WARN ][o.o.s.OpenSearchSecurityPlugin] [securityBwcCluster1-0] OpenSearch Security plugin installed but disabled. This can expose your configuration (including passwords) to the public.

After looking at the configuration settings, (Thanks for paired debugging with me @parasjain1) these test don't actually start up the security plugin at all, the following configuration line needs to be switched from false -> true.

security/bwc-test/build.gradle

Line 128 in 8063e1b

node.setting("plugins.security.disabled", "true")

However, after attempting to startup the node, copying configuration from SecureRestClientBuilder.java, there are still many errors coming in from the cluster:

» ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [securityBwcCluster1-0] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 4...0a
» io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 4..a
» at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1225) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]

These errors are happening because the cluster startup check is hardcoded to http, need to make this parameterized

WaitForHttpResource wait = new WaitForHttpResource("http", getFirstNode().getHttpSocketURI(), nodes.size());
[Source]

[peternied]

I've uploaded the stopping point I have on unblocking these scenarios - main...peternied:security:bwc-ssl @scrawfor99 This might be a good starting point.

[peternied]

I've created an issue that track the fundamental problem #3056 and resolution. Once that has been addressed we will circle back on this issue to make sure nothing else is coming up.

[stephen-crawford]

[Triage] This is the same topic as #3056, going to close this issue in favor of the actionable item breakdown.