aws shaded SDK is exporting netty dependencies

[steveloughran]

The aws-sdk-bundle, which is meant to be the shaded SDK, is now exporting netty

+- org.apache.hadoop:hadoop-aws:jar:3.2.0-SNAPSHOT:compile
   \- com.amazonaws:aws-java-sdk-bundle:jar:1.11.271:compile
      +- io.netty:netty-codec-http:jar:4.1.17.Final:compile
      +- io.netty:netty-codec:jar:4.1.17.Final:compile
      +- io.netty:netty-handler:jar:4.1.17.Final:compile
      +- io.netty:netty-buffer:jar:4.1.17.Final:compile
      +- io.netty:netty-common:jar:4.1.17.Final:compile
      +- io.netty:netty-transport:jar:4.1.17.Final:compile
      \- io.netty:netty-resolver:jar:4.1.17.Final:compile

1. Please stop it. The purpose of shading is to avoid version conflicts.
2. If we exclude these, what's going to break?

See: HADOOP-15264

[shorea]

Would renamespacing these work for you? Netty is used for the Kinesis Video client.

[steveloughran]

The reason we switched to the shaded SDK bundle in hadoop as transitive dependencies on things like jackson meant it was impossible to upgrade the AWS SDK versions. I'm afraid we expect the "shaded SDK" to be precisely that.

Can you confirm that it is only the Kinesis Video client. which adds the dependencies? In which case we can probably avoid reverting to 1.11.199, instead patching up our dependencies. However, that's precisely the game we've been trying to avoid "randomly excluding transient dependencies in the hope things work again"

[shorea]

Yes only Kinesis Video uses Netty at the moment. I'll add a mapping to the shade config to relocate netty like our other dependencies.

https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-bundle/pom.xml#L46

[steveloughran]

Thanks; we'll exclude the netty versions & not worry about Kinesis, as the only services directly used are: S3, STS, KMS, DDB; Some people may use the same package for Kinesis, but they can redeclare things. We'll have to add some notes about a JAR update process too: worth formalising

[steveloughran]

OK, we're cutting out these dependencies. We'll have to enhance our dependency update process to catch this sooner too.

[millems]

This has been fixed in the latest release.

[steveloughran]

thanks. Will check for it when we update the AWS SDK for a future hadoop release