[BUG] Security plugin fails to load locally on 1.3 (latest changes) branch

[saikaranam-amazon]

What is the bug?
Security plugin fails to load locally on 1.3 (latest changes) branch

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Build security plugin on 1.3 branch
2. Install and load the plugin on opensearch 1.3.4 version
3. Failures can be observed from the node logs

fatal error in thread [Thread-6], exiting
»  java.lang.NoClassDefFoundError: io/netty/channel/unix/UnixChannel
»  	at io.netty.handler.ssl.SslHandler.setOpensslEngineSocketFd(SslHandler.java:2142)
»  	at io.netty.handler.ssl.SslHandler.handlerAdded(SslHandler.java:1980)
»  	at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
»  	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
»  	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:181)
»  	at io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:152)
»  	at org.opensearch.security.ssl.http.netty.SecuritySSLNettyHttpServerTransport$SSLHttpChannelHandler.initChannel(SecuritySSLNettyHttpServerTransport.java:84)
»  	at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
»  	at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
»  	at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:938)
»  	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
»  	at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
»  	at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
»  	at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
»  	at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
»  	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
»  	at io.netty.channel.AbstractChannel$AbstractUnsafe.register(AbstractChannel.java:480)
»  	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87)
»  	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81)
»  	at io.netty.channel.MultithreadEventLoopGroup.register(MultithreadEventLoopGroup.java:86)
»  	at io.netty.bootstrap.ServerBootstrap$ServerBootstrapAcceptor.channelRead(ServerBootstrap.java:215)
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
»  	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97)
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
»  	at java.base/java.lang.Thread.run(Thread.java:829)
»  Caused by: java.lang.ClassNotFoundException: io.netty.channel.unix.UnixChannel
»  	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
»  	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
»  	at java.base/java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:904)
»  	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
»  	... 36 more

What is the expected behavior?
security plugin should be initialised properly and no issues should be seen.

What is your host/environment?

• Our CI is failing for security tests due to this issue: https://github.com/opensearch-project/cross-cluster-replication/runs/7277957246?check_suite_focus=true

[cliu123]

The netty version in OpenSearch core got upgraded 4 days ago, could the new version of netty have something causing this issue?

[zelinh]

@saratvemulapalli Could you maybe take a look at this and see if this is related to the OS core change?

[cwperks]

@saratvemulapalli @zelinh I am looking into this issue and discovered the root cause.

I upgraded from Netty 4.1.73.Final -> 4.1.78.Final in core last week to address a CVE and missed an underlying change that affects the security plugin. Netty introduced a new dependency to netty-handler on netty-transport-native-unix-common in this PR: netty/netty@bc085b0

The transitive dependency is not pulled in automatically and is not in core's module/transport-netty4/build.gradle dependencies section: https://github.com/opensearch-project/OpenSearch/blob/main/modules/transport-netty4/build.gradle#L53

Version 1.3 of the security plugin imports this module so the transitive dependency should be included. I am working on a fix now.

[peternied]

Thanks for picking this up @cwperks

[cliu123]

@cwperks Thanks for resolving the issue!

[saikaranam-amazon]

@cliu123 @cwperks
Do we know why only security plugin was affected due to this dependency? and
Curious to know as to why CI passed on 1.3 with the latest commit (without adding this dependency)?

Let's close this issue by attaching the above information.

[cliu123]

@saikaranam-amazon

Do we know why only security plugin was affected due to this dependency?

The dependency is only needed in SSLHandler that only security plugin has.

Curious to know as to why CI passed on 1.3 with the latest commit (without adding this dependency)?

Security plugin 1.3 does not reference the netty version of OpenSearch core. It specifies netty version in security plugin itself instead, so it is not affected by this missing dependency.

[saikaranam-amazon]

Thanks @cliu123
The CI failure (attached in the description) was observed while bringing up the nodes for Integration tests with security plugin. As this is the installation step, Does security CI not covering it?

[cliu123]

The Plugin Install workflow was supposed to catch this issue. The reason why it didn't catch needs to be investigated. @peternied has more context on this workflow.

[peternied]

@cliu123 That link is from a CI test against OpenSearch version 2.1 whereas the bug report is for version 1.3.

Reopening, as I think the fix for this issue needs to be backported to 1.3, @cwperks can you look into this?

[davidlago]

@cwperks was this ever backported to 1.3?

[cwperks]

@davidlago Yes it was: opensearch-project/OpenSearch#3861