Support for maven-shade-plugin

[mstevens83]

Because Android includes stale (or rather ancient) versions of a few popular open source libraries (e.g. Apache HttpClient & Apache Commons Codec) app developers may run into problems when they want to make use of more recent versions of these libraries (e.g. #479).
Thankfully there is a Maven plugin, called maven-shade-plugin, which makes dealing with this sort of situation easy (e.g. http://stackoverflow.com/a/16916552/1084488).
But sadly android-maven-plugin and maven-shade-plugin do not seem to be compatible.
By default the maven-shade-plugin attaches itself to the package phase, meaning it performs the "class relocation" at the wrong moment in the apk build cycle, namely after, instead of before, dexing. Consequently only the jar is affected, not the apk (meaning the app code will still try to reference the library with the clashing package name).
I was therefore hoping it would be possible to make future versions of android-maven-plugin aware of maven-shade-plugin such that the shading can be triggered at the correct moment.
This possibility has been discussed before in 2011 (see https://code.google.com/p/maven-android-plugin/issues/detail?id=170) and a preliminary patch was produced, but as far as I've been able to find out the effort was abandoned.
In my opinion the most elegant solution would be to introduce some kind of <executeShadePlugin>true</executeShadePlugin> parameter in the <configuration> which would make android-maven-plugin aware of maven-shade-plugin.
Or, if the 2 plugins are inherently incompatible (which may be the case, see comment below) than perhaps it would be best if android-maven-plugin was extended with built-in support for this kind of "class relocation", without actually using the shade plugin as such.

[mstevens83]

I did some further research and it seems that integration between maven-shade-plugin may actually have become at lot harder then it was with the 2011-era version of android-maven-plugin. With the current version the creation of a jar file happens in the package phase, whereas before it was during process-classes. This means the jar is only created after dexing (which happens in prepare-package), which is too late for the shading to affect the classes.
To complicate matters further it seems like maven-shade-plugin can only perform its "class relocation" magic on a jar file, not on unpackaged class files.
Still I hope a solution is or will be possible... Maybe by achieving the class relocation work with something else than the maven-shade-plugin?

[william-ferguson-au]

Why don't you change the phase in which the shade plugin executes?

On Thu, Oct 23, 2014 at 9:57 PM, Matthias Stevens [email protected]
wrote:

Because Android includes stale (or rather ancient) versions of a few
popular open source libraries (e.g. Apache HttpClient & Apache Commons
Codec) app developers may run into problems when they want to make use of
more recent versions of these libraries (e.g. #479
#479).
Thankfully there is a Maven plugin, called maven-shade-plugin, which
makes dealing with this sort of situation easy (e.g.
http://stackoverflow.com/a/16916552/1084488).
But sadly android-maven-plugin does not play along...
Because the maven-shade-plugin attaches to the package phase the
"shading" happens at the wrong moment in the apk build cycle, namely after,
instead of before, dexing. Consequently only the jar gets the shaded, not
the apk.
I was therefore hoping it would be possibly to make future versions of
android-maven-plugin aware of maven-shade-plugin such that the
shading can be triggered at the correct moment.
This possibility has been discussed before in 2011 (see
https://code.google.com/p/maven-android-plugin/issues/detail?id=170) and
a preliminary patch was produced, but as far as I've been able to sfind out
the effort was abandoned.
I my opinion the most elegant solution would be to introduce some kind of
true parameter in the
which would make android-maven-plugin aware of
maven-shade-plugin.

—
Reply to this email directly or view it on GitHub
#487.

[mstevens83]

I tried attaching it to process-classes but then it fails:

[INFO] --- maven-shade-plugin:2.3:shade (default) @ sapelli-collector-android ---
[ERROR] The project main artifact does not exist. This could have the following
[ERROR] reasons:
[ERROR] - You have invoked the goal directly from the command line. This is not
[ERROR]   supported. Please add the goal to the default lifecycle via an
[ERROR]   <execution> element in your POM and use "mvn package" to have it run.
[ERROR] - You have bound the goal to a lifecycle phase before "package". Please
[ERROR]   remove this binding from your POM such that the goal will be run in
[ERROR]   the proper phase.
[ERROR] - You removed the configuration of the maven-jar-plugin that produces the main artifact.

In other words, the shade plugin seems to require a jar file to work its magic on. I haven't found a way to make it perform the class relocation on unpackaged class files.
Moreover, if this were possible I would need to make sure the relocation is not only applied to classes of the project itself but also of dependencies. Because the library I am trying to shade for use on android is in fact a secondary dependency.

[william-ferguson-au]

You could always shade your libraries in a separate project and produce a
shaded lib that can be imported into your APK/AAR project.

On Thu, Oct 23, 2014 at 10:54 PM, Matthias Stevens <[email protected]

wrote:

I tried attaching it to process-classes but then it fails:

[INFO] --- maven-shade-plugin:2.3:shade (default) @ sapelli-collector-android ---
[ERROR] The project main artifact does not exist. This could have the following
[ERROR] reasons:
[ERROR] - You have invoked the goal directly from the command line. This is not
[ERROR] supported. Please add the goal to the default lifecycle via an
[ERROR] element in your POM and use "mvn package" to have it run.
[ERROR] - You have bound the goal to a lifecycle phase before "package". Please
[ERROR] remove this binding from your POM such that the goal will be run in
[ERROR] the proper phase.
[ERROR] - You removed the configuration of the maven-jar-plugin that produces the main artifact.

In other words, the shade plugin seems to require a jar file to work its
magic on. I haven't found a way to make it perform the class relocation on
unpackaged class files.

—
Reply to this email directly or view it on GitHub
#487 (comment)
.

[mstevens83]

True, but the consequence of that is that an important benefit of the shading plugin would be lost. Namely the fact that it transparently relocates classes without the programmer having to change imports accordingly.

For example, it allows one to package a jar in which org.apache.commons.codec becomes shaded.org.apache.commons.codec, but it also takes care of modifying any (other) class files in the jar which refer to relocated classes, even though they were compiled from unmodified source code (i.e. with imports pointing to org.apache.commons.codec).

But if, as you suggest, I use a separate project to perform the shading then the source code of my primary apk project will have to modified (i.e. change imports to point to shaded.org.apache.commons.codec).

The beauty of not having to modify source code is that it means a dependency clash would be solved entirely through build-time maven trickery, without bothering the (java) programmer. The shade plugin makes this possible for jar packaging and I was hoping this "magic" could be carried through to apk packaging.

[william-ferguson-au]

In an environment like Android that sounds really dangerous.

If you want to use a different version of a library then you should be
explicit about it. Some of the libraries that you have a dependency upon
may have real problems if you start injecting something other that the
version that ship with Android into them. However, if you rename them then
only that code that is explicitly configured to use them will. Which is a
lot safer.

On Thu, Oct 23, 2014 at 11:13 PM, Matthias Stevens <[email protected]

wrote:

True, but the consequence of that is that one of the benefits of the
shading plugin would be lost. Namely the fact that it transparently
relocates classes without the programmer having to change imports
accordingly.

For example, it allows one to package a jar in which
org.apache.commons.codec becomes shaded.org.apache.commons.codec, but it
also takes care of modifying any (other) class files in the jar which refer
to relocated classes, even though they were compiled from unmodified source
code (i.e. with imports pointing to org.apache.commons.codec).

But if, as you suggest, I use a separate project to perform the shading
then the source code of my primary apk project will have to modified
(i.e. change imports to point to shaded.org.apache.commons.codec).

The beauty of not having to modify source code is that it means a
dependency clash would be solved entirely through build-time maven
trickery, without bothering the (java) programmer. The shade plugin makes
this possible for jar packaging and I was hoping this "magic" could be
carried through to apk packaging.

—
Reply to this email directly or view it on GitHub
#487 (comment)
.

[mstevens83]

You are right about it being dangerous if other Android-specific dependencies also depend on the renamed dependency. In my situation this is not the case however.
But in fact there is another reason why your suggested solution (i.e. having a separate jar project to perform the shading) is not ideal. It turns out that in that case the apk produced by android-maven-plugin actually contains both the original and the shaded version of the library. Meaning the apk gets bigger than it has to be.
So at the very least android-maven-plugin should avoid this I think.

[mstevens83]

To clarify, this double inclusion happens in the following situation:
My apk project depends on my own library (jar) project, which in turns depends on commons-codec and shades it. The library project contains java code which actually uses commons-codec, but the apk project (currently) does not.
If you want proof of this double inclusion I can send you a jar file (produced by dex2jar from the resulting apk), in which both the shaded and the original commons-codec classes appear.

[mstevens83]

This double inclusion is very weird because in fact the shade plugin actually installs a "dependency-reduced" version of the library project pom file to the local m2 repo. This means the commons-codec dependency is in fact removed from the installed pom (which makes sense because the jar which is installed actually contains the shaded commons-codec classes).
So in fact my apk project shouldn't even "know" that my library depends on commons-codec.

[mstevens83]

I may need to mention that my apk and library project share the same parent.

[mstevens83]

Another side note, the dangerous situation you describe is not very plausible in the particular case of commons-codec because it is an internal Android library, i.e. not exposed in the Android SDK (unlike Apache HttpClient, which is exposed). So any Android app or library which needs commons-codec has to explicitly depend on/contain its own version of it in order to compile against the Android SDK.
The clash with Android's internal version only happens at runtime. For example when apk code tries to call a method which is not present in Android's antiquated internal commons-codec version:

10-13 16:22:39.361: I/dalvikvm(3776): Could not find method org.apache.commons.codec.binary.Hex.encodeHexString, referenced from method uk.ac.ucl.excites.sapelli.storage.db.sql.sqlite.types.SQLiteBlobColumn.sqlToLiteral

So even though it is absent at compile-time at runtime Android's internal commons-codec version annoyingly takes preference over the one included in the apk (and of course this is the reason I need some kind of shading to begin with).

[william-ferguson-au]

If you are getting commons-codec as well as your shaded commons-codec
included, then you have a problem with your dependencies or your config of
the shade-plugin.

On Thu, Oct 23, 2014 at 11:50 PM, Matthias Stevens <[email protected]

wrote:

This double inclusion is very weird in fact because the shade plugin
actually strips the dependency on commons-codec out of the pom file which
is installed to the local m2 repo (which makes sense because the jar which
is installed actually contains the shaded commons-codec classes).
So in fact my apk project shouldn't even "know" that my library depends on
commons-codec.

—
Reply to this email directly or view it on GitHub
#487 (comment)
.

[mstevens83]

I still think it is somehow related to android-maven-plugin. Here is a minimal working example:

Parent pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>test</groupId>
  <artifactId>parent</artifactId>
  <version>1.0.0</version>
  <packaging>pom</packaging>
  <modules>
    <module>shaded-commons-codec</module>
    <module>helloflashlight</module>
  </modules>
</project>

Library project pom:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>test</groupId>
        <artifactId>parent</artifactId>
        <version>1.0.0</version>
    </parent>

    <artifactId>shaded-commons-codec</artifactId>
    <packaging>jar</packaging>

    <dependencies>
        <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec</artifactId>
            <version>[1.9,)</version>
        </dependency>
    </dependencies>
    <build>
        <finalName>${project.artifactId}</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.apache.commons.codec</pattern>
                                    <shadedPattern>shaded.org.apache.commons.codec</shadedPattern>
                                </relocation>
                            </relocations>
                            <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                            <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Apk project, this is just the helloflashlight project taken from android-maven-plugin samples, only modifications are in the pom (added parent and the shaded-commons-codec dependency):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>test</groupId>
        <artifactId>parent</artifactId>
        <version>1.0.0</version>
    </parent>

  <artifactId>helloflashlight</artifactId>
  <packaging>apk</packaging>
  <name>HelloFlashlight</name>

  <dependencies>
    <dependency>
      <groupId>com.google.android</groupId>
      <artifactId>android</artifactId>
      <version>4.1.1.4</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>test</groupId>
      <artifactId>shaded-commons-codec</artifactId>
      <version>1.0.0</version>
    </dependency>
  </dependencies>

  <build>
    <sourceDirectory>src</sourceDirectory>
    <finalName>${project.artifactId}</finalName>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>com.jayway.maven.plugins.android.generation2</groupId>
          <artifactId>android-maven-plugin</artifactId>
          <version>4.0.0-rc.1</version>
          <extensions>true</extensions>
        </plugin>
      </plugins>
    </pluginManagement>
    <plugins>
      <plugin>
        <groupId>com.jayway.maven.plugins.android.generation2</groupId>
        <artifactId>android-maven-plugin</artifactId>
        <configuration>
          <sdk>
            <!-- platform as api level (api level 16 = platform 4.1)-->
            <platform>16</platform>
          </sdk>
          <androidManifestFile>${project.basedir}/AndroidManifest.xml</androidManifestFile>
          <resourceDirectory>${project.basedir}/res</resourceDirectory>
        </configuration>
      </plugin>
    </plugins>
  </build>

</project>

When running "mvn install" on the parent pom (which in turn installs both modules) I and up with an apk which contains both the original and the shaded commons-codec.
However this does not happen if the apk and library project do not share a parent. But I don't see how the fact that they do justifies what is happening...

[mstevens83]

So as my example shows, when the apk and library project share a parent android-maven-plugin somehow becomes aware of the fact that the library originally did depend on commons-codec, even though the library jar and pom which are installed in the local repository do not.

[mosabua]

Its supposed to do that. But in your dependency declaration to the shaded jar you just add an exclusion to the original. Or you use provided scope in the project where you are shading it..

[mstevens83]

Why exactly is it supposed to do that?

This means android-maven-plugin effectively "invents" a dependency which shouldn't be there (given that the library, as-installed to the repo, includes rather then depends on commons-codec) and which is not even listed when I run mvn dependency:tree on the apk project.

Also note that android-maven-plugin doesn't do this if the apk and library project do not share their parent. So at the very least the behaviour is inconsistent.

[mosabua]

The pom for the shaded jar declares a dependency to commons-codec... and as such it is a transitive dependency.. which needs to be pulled in. Just add the exclusion like I mentioned and it will work.

[mstevens83]

The suggested exclusion does indeed work (thanks!), but I'm trying to convince you guys there is an actual bug (or at least an inconsistency) here. Perhaps you didn't read the entire thread.
Please consider this:

in fact the shade plugin actually installs a "dependency-reduced" version of the library project pom file to the local m2 repo. This means the commons-codec dependency is in fact removed from the installed pom (which makes sense because the jar which is installed actually contains the shaded commons-codec classes [and no longer depends on commons-codec as an external jar]).
So in fact my apk project shouldn't even "know" that my library [originally] depends on commons-codec.

So really, the behaviour is inconsistent: when the apk and library projects do not share a parent then the installed library pom is used (this is the "dependency-reduced" one, which does not depend on commons-codec, not even transitively, as per the shading), but when the projects do share a parent then original library pom is somehow found and used, causing android-maven-plugin to think the library still depends on commons-codec, resulting in the double inclusion.
As I noted above only android-maven-plugin is fooled because dependency:tree does not list commons-code as a dependency of the library, whether the projects share their parent or not (i.e. it always considers the installed library pom, as I think it should).

[mstevens83]

To clarify, this is the "depenency-reduced" library pom, as installed to the local repo:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <parent>
    <artifactId>parent</artifactId>
    <groupId>test</groupId>
    <version>1.0.0</version>
    <relativePath>../../pom.xml</relativePath>
  </parent>
  <modelVersion>4.0.0</modelVersion>
  <artifactId>shaded-commons-codec</artifactId>
  <build>
    <finalName>${project.artifactId}</finalName>
    <plugins>
      <plugin>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.3</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <relocations>
                <relocation>
                  <pattern>org.apache.commons.codec</pattern>
                  <shadedPattern>shaded.org.apache.commons.codec</shadedPattern>
                </relocation>
              </relocations>
              <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
              <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

As you can see it does not depend on commons-codec, due to the work of the shading plugin.

[mstevens83]

By the way, while your first suggestion (adding an exclusion to the original to the dependency declaration to the shaded jar) works the second one (using provided scope in the project where the original is shaded) does not. In the second case the shade plugin failed to do its work and I get a jar which only contains META-INF.

[mosabua]

Hm ... I made a wrong assumption about the shade plugin working with provided scope. Sorry about that.

In terms of the pom... if you have it in a multi module project the pom within the project is used and not what the shade plugin cooks up and pushes into the local repo. If you pull it apart into a separate project that publishes to your repo .. then it should work without exclusion..

[mstevens83]

Ok thanks for explaining.
I accept that this shading business is not a priority for you (hence the closing of the issue), but I still think it would be nice to have a way in which android-maven-plugin itself can prevent library clashes. Because the problem I'm trying to solve only happens on Android, but in my actual situation (not the mock-up I posted here) the library project (which is now polluted with all this shading stuff) is also used by j2se desktop/server projects, which do not need the shading. So ideally I wouldn't like to solve the problem where it appears, i.e. when packaging an apk for use on Android, which is why I posted this issue to begin with.
But anyway, think you both for your help so far.

[mosabua]

Problem is that we just use the dependency resolving from Maven and dont to much custom stuff. Also how would we know what is shaded and what not. Frankly I have no clear idea how this could be resolved without a lot of effort and considering that there is an easy solution I just dont see the need.

[mstevens83]

I found out that using "runtime" scope on the commons-codec dependency in my library project does not prevent the shade plugin from doing its work (unlike with the "provided" scope), but (like the provided scope) also works as a way to prevent the inclusion of the original commons-codec in the final apk.

So for other people face this problem, here is my final solution:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

  <!--
    *************************************************************
     Rationale for this "shaded" version of Apache Commons Codec
    *************************************************************

    Context:
      Android includes an outdated version (1.3) of commons-codec as an internal library.
      This library is not exposed in the Android SDK so app developers who want to rely on
      commons-codec need to treat it as an addition dependency and include it in the APK of
      their app. However, at runtime Android will always favour its internal version of the
      library which causes trouble when app code tries to call methods that don't exist in
      that outdated version but do exist in the version the developer expected to be using.

    Solution:
      After experimenting with many different variations the current (and final) solution
      to this problem is implemented in this project and does not require any hacks of changes
      in the other projects, expect for declaring dependency on commons-codec-shaded
      (i.e. this project) instead of the original commons-codec.

      What we do here is take the "original" commons-codec library (currently version 1.9) and
      use the maven-shade-plugin to "shade" it, which means we modify the package name of the
      library (both in the compiled classes and the sources jar) in order to avoid the clash
      with Android's version. The package name changes from "org.apache.commons.codec" to
      "shaded.org.apache.commons.codec". The result is published to the local Maven repository
      for other projects to use by simple dependency declaration on this project.
      Because we only apply the shading to commons-codec itself (and not to other classes
      using it; which is possible using the shade plug-in but doesn't work in combination with
      android-maven-plugin) any client classes which make use of commons-codec will have to
      import the new "shaded" package name instead of the old one.

      Note on the "runtime" scope on the commons-codec dependency declaration below:
        If we do not set a scope other than the default "compile", then an apk-producing project
        which is being built with Maven, as part of (rather than on its own) the building of
        a parent pom which also includes this project as a module, will pull both(!) the shaded
        and the original commons-codec library into the apk (making it larger than necessary).

        One way around this is to: (1) make the shade plugin generate a "dependency-reduced"
        pom.xml (in which the commons-codec dependency is removed) for publication to the local
        repository, and (2) always build the apk separately or removing this project's module from
        the parent (because otherwise Maven will still find and use the original pom.xml, and the
        original commons-codec will still be considered a dependency to be pulled into the apk).
        This is not desirable because it does away with the ability to build all projects with a
        single command (i.e. building the parent pom).

        Another way is to explicitly exclude the original commons-codec as part of the dependency
        declaration on this project:
          <dependency>
            <groupId>commons-codec</groupId>
            <artifactId>commons-codec-shaded</artifactId>
            <version>1.9</version>
            <exclusions>
              <exclusion>
                <groupId>commons-codec</groupId>
                <artifactId>commons-codec</artifactId>
              </exclusion>
            </exclusions> 
          </dependency>
        This is not desirable either because it looks weird/confusing and more importantly it spreads
        the solution to the library clash problem to a 2nd project (in the sense that we need to do
        more than just declare dependency on commons-codec-shaded instead of the original commons-codec).

        A third way could have been to declare the scope a "provided", which makes sense because
        this project essentially "provides" commons-codec and it would some the original library from
        being pulled into the apk as well. Yet this doesn't work because the shade plugin itself also
        ignores provided dependencies (so it wouldn't do the "shading" at all).
        The current solution of using a "runtime" scope does work however: the shade plugin does not
        ignore "runtime" dependencies (so shading happens), and the runtime scope has the same effect
        as the provided scode at apk package time (i.e. the original library is not being pulled in).

    Issue on android-maven-plugin github which I posted to discuss all this:
      https://github.com/jayway/maven-android-plugin/issues/487
  -->

  <groupId>commons-codec</groupId>
  <artifactId>commons-codec-shaded</artifactId>
  <name>Apache Commons Codec [shaded]</name>
  <!-- the version of this project specifies the Apache Commons Codec version which will
       be used, it must therefore match an existing (and preferably current) version. -->
  <version>1.9</version>
  <packaging>jar</packaging>

  <developers>
    <developer><!-- for shaded version only -->
      <name>Matthias Stevens</name>
    </developer>
  </developers>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <shaded.package.prefix>shaded.</shaded.package.prefix>
  </properties>

  <dependencies>
    <dependency>
      <groupId>commons-codec</groupId>
      <artifactId>commons-codec</artifactId>
      <!-- the project version specifies the commons-codec version to use -->
      <version>${project.version}</version>
      <!-- Don't change (reason explained above) -->
      <scope>runtime</scope>
    </dependency>
  </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
    <plugins>
      <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.apache.commons.codec</pattern>
                            <shadedPattern>${shaded.package.prefix}org.apache.commons.codec</shadedPattern>
                                </relocation>
                            </relocations>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <createSourcesJar>true</createSourcesJar>
                            <shadeSourcesContent>true</shadeSourcesContent>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

[william-ferguson-au]

Isn't the correct solution to this to set createDependencyReducedPom=true
and include the reduced-pom-artifact as your dep in your APK?
That way the APK and AMP know nothing about commons-codec until you are
also telling your APK about it via another dep.

On Fri, Oct 24, 2014 at 11:12 PM, Matthias Stevens <[email protected]

wrote:

I found out that using "runtime" scope on the commons-codec dependency in
my library project does not prevent the shade plugin from doing its work
(unlike with the "provided" scope), but (like the provided scope) also
works as a way to prevent the inclusion of the original commons-codec in
the final apk.

So for other people face this problem, here is my final solution:

4.0.0

commons-codec
commons-codec-shaded
Apache Commons Codec [shaded]

1.9
jar

Matthias Stevens UTF-8 UTF-8 shaded. commons-codec commons-codec ${project.version} runtime

<build>
    <finalName>${project.artifactId}</finalName>
<plugins>
  <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.3</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>org.apache.commons.codec</pattern>
                        <shadedPattern>${shaded.package.prefix}org.apache.commons.codec</shadedPattern>
                            </relocation>
                        </relocations>
                        <createDependencyReducedPom>false</createDependencyReducedPom>
                        <createSourcesJar>true</createSourcesJar>
                        <shadeSourcesContent>true</shadeSourcesContent>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

—
Reply to this email directly or view it on GitHub
#487 (comment)
.

[mosabua]

Sort of.. problem is that the dependency reduced pom is created as part of the build. And the pom creating that has the dependency in it and in a multi module build it participates in the reactor. We talked about this in the last Maven dev hangout and Igor mentioned that using the scope optional might work... worth a try @mstevens83

[mstevens83]

mosabua: you mean optional scope instead of provided (or runtime, as I did above?).

[mstevens83]

The last pom.xml posted above turned out not to be the final solution after all.
I found out I still had to put the exclusion of the original commons-codec at the point of dependency on shade-commons-codec. Although the runtime scope (I never tried it with optional) avoided the double inclusion (of both shaded-commons-codec and the original one) in my APKs, my IDE (Eclipse) still considered both libraries as dependencies, which still left a risk of a programmer using the "unshaded" commons-codec.
Because I really dislike the exclusion at the point of dependency I ended up doing it in yet another way...

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>commons-codec</groupId>
    <artifactId>commons-codec-shaded</artifactId>
    <name>Apache Commons Codec (shaded)</name>
    <!-- The version of this project specifies the Apache Commons Codec version which will
         be used, it must therefore match an existing (and preferably current) version. -->
    <version>1.9</version>
    <packaging>jar</packaging>

    <!--
      *************************************************************
       Rationale for this "shaded" version of Apache Commons Codec
      *************************************************************

      Context:
        Android includes an outdated version (1.3) of commons-codec as an internal library.
        This library is not exposed in the Android SDK so app developers who want to rely on
        commons-codec need to treat it as an addition dependency and include it in the APK of
        their app. However, at runtime Android will always favour its internal version of the
        library which causes trouble when app code tries to call methods that don't exist in
        that outdated version but do exist in the version the developer expected to be using.

      Solution:
        After experimenting with many different variations the current (and final) solution
        to this problem is implemented in this project and does not require big hacks or changes
        in the other projects, expect for declaring dependency on commons-codec-shaded
        (i.e. this project) instead of the original commons-codec, and
        explicitly excluding the dependency on the original one.

        What we do here is take the "original" commons-codec library (currently version 1.9) and
        use the maven-shade-plugin to "shade" it, which means we modify the package name of the
        library (both in the compiled classes and the sources jar) in order to avoid the clash
        with Android's version. The package name changes from "org.apache.commons.codec" to
        "shaded.org.apache.commons.codec". The result is published to the local Maven repository
        for other (sapelli) projects to use by simple dependency declaration on this project.
        Because we only apply the shading to commons-codec itself (and not to other classes
        using it; which is possible using the shade plug-in but doesn't work in combination with
        android-maven-plugin) any client classes which make use of commons-codec will have to
        import the new "shaded" package name instead of the old one.

      Issue on android-maven-plugin github which I posted to discuss all this:
        https://github.com/jayway/maven-android-plugin/issues/487
    -->

    <description>
     The Apache Commons Codec package contains simple encoder and decoders for
     various formats such as Base64 and Hexadecimal.  In addition to these
     widely used encoders and decoders, the codec package also maintains a
     collection of phonetic encoding utilities.
    </description>
    <url>http://commons.apache.org/proper/commons-codec/</url>
    <organization>
        <name>The Apache Software Foundation</name>
        <url>http://www.apache.org/</url>
    </organization>
    <licenses>
        <license>
            <name>The Apache Software License, Version 2.0</name>
            <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
            <distribution>repo</distribution>
        </license>
    </licenses>

    <contributors>
        <contributor>
            <name>Matthias Stevens</name>
            <email>m.stevens {at} ucl.ac.uk</email>
            <roles>
                <role>Shading for use on Android</role>
            </roles>
        </contributor>
        <!-- see commons-codec:commons-codec pom for list of original contributors & developers -->
    </contributors>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <maven.compiler.source>1.6</maven.compiler.source>
        <maven.compiler.target>1.6</maven.compiler.target>
        <commons-codec-package>org.apache.commons.codec</commons-codec-package>
        <shading.prefix>shaded</shading.prefix>
        <shaded-commons-codec-package>${shading.prefix}.${commons-codec-package}</shaded-commons-codec-package>
        <commons-codec-src-folder>${project.build.directory}/commons-codec-src</commons-codec-src-folder>
        <commons-codec-res-folder>${project.build.directory}/commons-codec-res</commons-codec-res-folder>
        <manifest.path>${project.build.directory}/MANIFEST.MF</manifest.path>
        <!-- plugin versions -->
        <dependency-plugin-version>2.9</dependency-plugin-version>
        <compiler-plugin-version>3.2</compiler-plugin-version>
        <antrun-plugin-version>1.7</antrun-plugin-version>
        <jar-plugin-version>2.5</jar-plugin-version>
        <source-plugin-version>2.4</source-plugin-version>
        <shade-plugin-version>2.3</shade-plugin-version>
        <bundle-plugin-version>2.5.3</bundle-plugin-version>
        <!-- taken/modified from: http://svn.apache.org/repos/asf/commons/proper/commons-parent/trunk/pom.xml -->
        <commons.osgi.symbolicName>${shaded-commons-codec-package}</commons.osgi.symbolicName>
        <commons.osgi.export>${shaded-commons-codec-package}.*;version=${project.version};-noimport:=true</commons.osgi.export>
        <commons.osgi.import>*</commons.osgi.import>
        <commons.osgi.dynamicImport />
        <commons.osgi.private />
    </properties>

    <build>
        <finalName>${project.artifactId}</finalName>
        <sourceDirectory>${commons-codec-src-folder}</sourceDirectory>      
        <resources>
            <resource>
                <!-- txt files in shaded\org\apache\commons\codec\language\bm -->
                <directory>${commons-codec-res-folder}</directory>
                <includes>
                    <include>${shading.prefix}/**/*.txt</include>
                </includes>
            </resource>
            <resource>
                <!-- LICENSE & NOTICE files -->
                <directory>${commons-codec-res-folder}/META-INF</directory>
                <targetPath>META-INF</targetPath>
                <includes>
                    <include>*.txt</include>
                </includes>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <!-- fetch & unpack commons-codec sources and resources -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>${dependency-plugin-version}</version>
                <executions>
                    <execution>
                        <id>unpack_commons-codec_sources_and_resources</id>
                        <phase>process-sources</phase>
                        <goals>
                            <goal>unpack</goal>
                        </goals>
                        <configuration>
                            <artifactItems>
                                <!-- commons-codec sources -->
                                <artifactItem>
                                    <groupId>commons-codec</groupId>
                                    <artifactId>commons-codec</artifactId>
                                    <!-- the project version specifies the commons-codec version to use: -->
                                    <version>${project.version}</version>
                                    <classifier>sources</classifier>
                                    <overWrite>true</overWrite>
                                    <excludes>**/*.txt,META-INF/*</excludes>
                                    <outputDirectory>${commons-codec-src-folder}</outputDirectory>
                                </artifactItem>
                                <!-- commons-codec resources (in package) -->
                                <artifactItem>
                                    <groupId>commons-codec</groupId>
                                    <artifactId>commons-codec</artifactId>
                                    <!-- the project version specifies the commons-codec version to use: -->
                                    <version>${project.version}</version>
                                    <classifier>sources</classifier>
                                    <overWrite>true</overWrite>
                                    <includes>org/**/*.txt</includes>
                                    <!-- apply shading: -->
                                    <outputDirectory>${commons-codec-res-folder}/${shading.prefix}</outputDirectory>
                                </artifactItem> -->
                                <!-- commons-codec resources (in META-INF) -->
                                <artifactItem>
                                    <groupId>commons-codec</groupId>
                                    <artifactId>commons-codec</artifactId>
                                    <!-- the project version specifies the commons-codec version to use: -->
                                    <version>${project.version}</version>
                                    <classifier>sources</classifier>
                                    <overWrite>true</overWrite>
                                    <includes>META-INF/*.txt</includes>
                                    <outputDirectory>${commons-codec-res-folder}</outputDirectory>
                                </artifactItem> -->
                            </artifactItems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- compile commons-codec sources -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${compiler-plugin-version}</version>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <version>${jar-plugin-version}</version>
                <executions>
                    <execution>
                        <!-- jar unshaded classes (& resources) -->
                        <id>jar-unshaded</id>
                        <phase>package</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                    <execution>
                        <!-- rejar shaded classes (& resources), with proper manifest partially generated by bundle plugin -->
                        <id>jar-shaded</id>
                        <!-- runs after bundle plugin has done its work to generate bundle manifest -->
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                        <configuration>
                            <archive>
                                <manifestFile>${manifest.path}</manifestFile>
                                <manifestEntries>
                                    <Specification-Title>${project.name}</Specification-Title>
                                    <Specification-Version>${project.version}</Specification-Version>
                                    <Specification-Vendor>${project.organization.name}</Specification-Vendor>
                                    <Implementation-Title>${project.name}</Implementation-Title>
                                    <Implementation-Version>${project.version}</Implementation-Version>
                                    <Implementation-Vendor>${project.organization.name}</Implementation-Vendor>
                                    <Implementation-Vendor-Id>org.apache</Implementation-Vendor-Id>
                                    <Implementation-Build>${implementation.build}</Implementation-Build>
                                    <X-Compile-Source-JDK>${maven.compiler.source}</X-Compile-Source-JDK>
                                    <X-Compile-Target-JDK>${maven.compiler.target}</X-Compile-Target-JDK>
                                </manifestEntries>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- attach sources jar -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>${source-plugin-version}</version>
                <configuration>
                    <archive>
                        <manifest>
                            <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
                            <addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <!-- jar unshaded sources -->
                        <id>attach-unshaded-sources</id>
                        <!-- <phase>package</phase> (default) -->
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                    <execution>
                        <!-- rejar shaded sources -->
                        <id>attach-shaded-sources</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>jar</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- apply the shading to main jar and sources jar -->
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>${shade-plugin-version}</version>
                <executions>
                    <execution>
                        <id>shading-main-jar-and-sources-jar</id>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <!-- (not needed as it is the one and only artifact/dependency)
                            <artifactSet> 
                                <includes>
                                    <include>commons-codec:*</include>
                                </includes>
                            </artifactSet>
                            -->
                            <relocations>
                                <relocation>
                                    <pattern>${commons-codec-package}</pattern>
                                    <shadedPattern>${shaded-commons-codec-package}</shadedPattern>
                                </relocation>
                            </relocations>
                            <createDependencyReducedPom>false</createDependencyReducedPom>
                            <!-- (only needed when dependency reduced pom is generated)
                            <dependencyReducedPomLocation>${project.build.directory}/dependency-reduced-pom.xml</dependencyReducedPomLocation> 
                            <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope> 
                            <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                            -->
                            <createSourcesJar>true</createSourcesJar>
                            <shadeSourcesContent>true</shadeSourcesContent>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>${antrun-plugin-version}</version>
                <executions>
                    <execution>
                        <!-- unpack shaded classes and sources for manifest generation and re-jarring -->
                        <id>post-shading-tasks</id>
                        <phase>package</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <target>
                                <!-- Unjar shaded classes for generation of manifest -->
                                <echo>Deleting unshaded classes...</echo>
                                <delete dir="${project.build.directory}/classes"/>
                                <echo>Unjarring shaded main jar...</echo>
                                <unzip src="${project.build.directory}/${project.artifactId}.jar" dest="${project.build.directory}/classes"/>
                                <!-- delete to prevent dual inclusion in new main jar -->
                                <delete dir="${project.build.directory}/classes/META-INF/maven"/>
                                <!-- Unjar shaded sources -->
                                <echo>Deleting unshaded sources...</echo>
                                <delete dir="${commons-codec-src-folder}"/>
                                <echo>Unjarring shaded sources jar...</echo>
                                <unzip src="${project.build.directory}/${project.artifactId}-sources.jar" dest="${commons-codec-src-folder}"/>
                                <!-- delete to prevent dual inclusion in new sources jar -->
                                <delete dir="${commons-codec-src-folder}/META-INF"/>
                            </target>
                        </configuration>
                    </execution>
                    <execution>
                            <id>delete-orginals</id>
                            <phase>verify</phase>
                            <goals>
                                <goal>run</goal>
                            </goals>
                            <configuration>
                                <target>
                                    <echo>Deleting unshaded jar files...</echo>
                                    <delete>
                                        <fileset dir="${project.build.directory}" includes="**/original-*.jar" />
                                    </delete>
                                </target>
                            </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <!-- taken/modified from: http://svn.apache.org/repos/asf/commons/proper/commons-parent/trunk/pom.xml -->
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>${bundle-plugin-version}</version>
                <configuration>
                    <archive>
                        <forced>true</forced>
                    </archive>
                    <excludeDependencies>true</excludeDependencies>
                    <manifestLocation>${project.build.directory}</manifestLocation>
                    <instructions>
                        <!-- stops the "uses" clauses being added to "Export-Package" manifest entry -->
                        <_nouses>true</_nouses>
                        <!-- Stop the JAVA_1_n_HOME variables from being treated as headers by Bnd -->
                        <_removeheaders>JAVA_1_3_HOME,JAVA_1_4_HOME,JAVA_1_5_HOME,JAVA_1_6_HOME,JAVA_1_7_HOME,JAVA_1_8_HOME</_removeheaders>
                        <Bundle-SymbolicName>${commons.osgi.symbolicName}</Bundle-SymbolicName>
                        <Export-Package>${commons.osgi.export}</Export-Package>
                        <Private-Package>${commons.osgi.private}</Private-Package>
                        <Import-Package>${commons.osgi.import}</Import-Package>
                        <DynamicImport-Package>${commons.osgi.dynamicImport}</DynamicImport-Package>
                        <Bundle-DocURL>${project.url}</Bundle-DocURL>
                    </instructions>
                </configuration>
                <executions>
                    <execution>
                        <id>bundle-manifest</id>
                        <!-- runs after the unjarring of the shaded classes -->
                        <phase>integration-test</phase><!--  default is: process-classes -->
                        <goals>
                            <goal>manifest</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

What happsn here is basically: fetch the commons-codec sources, compile, package classes and source, shade both, unpack both, repack with bundle manifest.
Because the project has no dependencies I no longer need to excluded commons-codec in the places where I uses shaded-commons-codec.

Quite a fun maven learning experience this was :-).

[mosabua]

Awesome hack... how about a pull request with this in an asciidoc formatted file in src/site/asciidoc as an example doc ;-)