Added <skipDependencies> option to ApkMojo and DexMojo

[lyubomyr-shaydariv]

Problem: Use Java 8 libraries in Android applications.

The story

I have a small set of minilibs I have written in Java 6 to share them between "regular Java", GWT and Android applications. This works fine, but Java 8 is here, so I decided to migrate all of my internal projects to Java 8 to use Java 8 language features (lambda expressions, method references, etc), and not JDK 1.8 new features like Stream API or so. I succeeded with not yet released GWT 2.8.0 to use Java 8 language level libraries (source code compatiblity), but couldn't make this approach work for the Android environment with android-maven-plugin due to incompatible bytecode version (54). First I tried to use retrolambda-maven-plugin hoping it helps, but then I figured out that it can only process ./target/classes directory.

Then I investigated the DEBUG-level logs generated by the android-maven-plugin: it currently tries to include repository-based libraries that are not yet instrumented with retrolambda-maven-plugin.

01: $JAVA_HOME/jre/bin/java
02: -Xmx1024M
03: -jar "$ANDROID_HOME/sdk/build-tools/android-4.4/lib/dx.jar"
04: --dex
05: --output=$BUILD_DIRECTORY/classes.dex
06: $BUILD_DIRECTORY/classes
07: $M2_REPO/foo1/bar1/0.1-J8-SNAPSHOT/bar1-0.1-J8-SNAPSHOT.jar
08: $M2_REPO/foo2/bar2/0.1-J8-SNAPSHOT/bar2-0.1-J8-SNAPSHOT.jar
09: $M2_REPO/foo3/bar3/0.1-J8-SNAPSHOT/bar3-0.1-J8-SNAPSHOT.jar

This led me an idea of obtaining all application dependencies to the ./target/classes directory and then instrumenting them with retrolambda-maven-plugin to let android-maven-plugin DEX-ify the local classes only that are supposed to include all the dependencies locally. Simply speaking, let the dx tool invoke the previous snippet without lines 07, 08, and 09. However, I couldn't find a way to exclude dependencies from the android-maven-plugin workflow.

What was done

Thus, this pull request has a new option for ApkMojo and DexMojo called skipDependencies that allows to ignore any JAR-files coming from the repo and configure the dx tool to accept the ./target/classes directory only. An example workflow I made my Android demo application use Java 8 libraries:

• maven-compiler-plugin to enable source and target 1.8 language support;
• maven-dependency-plugin with goal set to unpack-dependencies to obtain all classes and resources in ./target/classes;
• retrolambda-maven-plugin to instrument Java 8 byte code;
• android-maven-plugin with the new <skipDependencies>true</skipDependencies> to process local ./target/classes only.

A portion from pom.xml I used to test my application:

    <plugin>
        <groupId>com.simpligility.maven.plugins</groupId>
        <artifactId>android-maven-plugin</artifactId>
        <version>4.2.2-SNAPSHOT</version>
        <extensions>true</extensions>
        <configuration>
            <androidManifestFile>${project.basedir}/src/main/android/AndroidManifest.xml</androidManifestFile>
            <assetsDirectory>${project.basedir}/src/main/android/assets</assetsDirectory>
            <resourceDirectory>${project.basedir}/src/main/android/res</resourceDirectory>
            <sdk>
                <platform>19</platform>
            </sdk>
            <undeployBeforeDeploy>true</undeployBeforeDeploy>
            <proguard>
                <skip>true</skip>
                <config>${project.basedir}/proguard.conf</config>
            </proguard>
            <skipDependencies>true</skipDependencies> <!-- the new one -->
        </configuration>
    </plugin>

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>2.10</version>
        <executions>
            <execution>
                <phase>process-classes</phase>
                <goals>
                    <goal>unpack-dependencies</goal>
                </goals>
                <configuration>
                    <includeScope>runtime</includeScope>
                    <outputDirectory>${project.build.directory}/classes</outputDirectory>
                </configuration>
            </execution>
        </executions>
    </plugin>

    <plugin>
        <groupId>net.orfjackal.retrolambda</groupId>
        <artifactId>retrolambda-maven-plugin</artifactId>
        <version>1.8.1</version>
        <executions>
            <execution>
                <phase>process-classes</phase>
                <goals>
                    <goal>process-main</goal>
                    <goal>process-test</goal>
                </goals>
            </execution>
        </executions>
    </plugin>

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
            <source>1.8</source>
            <target>1.8</target>
        </configuration>
    </plugin>

[william-ferguson-au]

Sounds like a reasonable use case (nicely explained - thank you).
And the code looks good.

+1 from me. @mosabua @Shusshu ?

[mosabua]

Should it somehow also be used in the lifecycle participant? Otherwise I am fine with this.. although I would think the approach to be more of a hack ;-)

[william-ferguson-au]

Hmm good point.
What happens when your dependencies included an AAR and a Java8 lib?

It's always a hack to support new mechanics before they are available in
the base platform.
ᐧ

On Wed, May 20, 2015 at 5:00 AM, Manfred Moser [email protected]
wrote:

Should it somehow also be used in the lifecycle participant? Otherwise I
am fine with this.. although I would think the approach to be more of a
hack ;-)

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[Shusshu]

Yep, very specific use case. I'd like to see some documentation on the site too. That would be helpful for anyone in the same situation

[lyubomyr-shaydariv]

Yeah, it's more like a hack, because my intention to use Java 8 libraries might be resulted into a simplier solution than I suggested, but this is currently the only way I know to make it work. My home projects are split into a few dozens of modules and these modules are mostly not yet ported to Java 8 keeping source and binary compatibility with Android and GWT respectively. A few days ago I succeeded with not yet released GWT 2.8.0 that finally has Java 8 language features support, and thus I thought I could try to migrate my Android test apps to Java 8 too. Initially I thought that retrolambda-maven-plugin might cover my needs, but I didn't know it can't instrument dependency classes. This is the only reason of maven-dependency-plugin being here. Perhaps this workflow can be helpful for someone else too for other cases like bytecode instrumenting, but yeah it looks like a sort of a very specific case if my initial question at S.O. still does not have answers. And this makes me think that no one uses Java 8 libraries in the Android world, however Retrolambda is a well-known tool. Or I just went a wrong way...

[william-ferguson-au]

I would say that very few people are using Java8 mechanisms in the Android
workld precisely because of the pain involved in doing so.

Your solution only works if ALL of the dependencies of the APK are Java
libs. As soon as one of them is an AAR (which is extremeley common - eg
AppCompat etc) then it will fail because the jars contained in the AAR deps
will be missing.
ᐧ

On Wed, May 20, 2015 at 3:16 PM, Lyubomyr Shaydariv <
[email protected]> wrote:

Yeah, it's more like a hack, because my intention to use Java 8 libraries
might be resulted into a simplier solution than I suggested, but this is
currently the only way I know to make it work. My home projects are split
into a few dozens of modules and these modules are mostly not yet ported to
Java 8 keeping source and binary compatibility with Android and GWT
respectively. A few days ago I succeeded with not yet released GWT 2.8.0
that finally has Java 8 language features support, and thus I thought I
could try to migrate my Android test apps to Java 8 too. Initially I
thought that retrolambda-maven-plugin might cover my needs, but I didn't
know it can't instrument dependency classes. This is the only reason of
maven-dependency-plugin being here. Perhaps this workflow can be helpful
for someone else too for other cases like bytecode instrumenting, but yeah
it looks like a sort of a very specific case if my initial question at S.O.
still does not have a nswers. And this makes me think that no one uses Java
8 libraries in the Android world, however Retrolambda is a well-known tool.
Or I just went a wrong way...

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[lyubomyr-shaydariv]

@william-ferguson-au
As far as I understand, you mean the ApkMojo only? If so, perhaps does it mean that <skipDependencies> is helpful in cases like mine, but there also should be more precise configuration options letting the dependency artifacts to be excluded or included individually? Let's say, something like:

<skipDependencies>true</skipDependencies>
<!-- supposed to have higher priority than skipDependencies -->
<includeArtifacts>
    <includeArtifact>foo:bar</includeArtifact>
</includeArtifacts>

and

<excludeArtifacts>
    <excludeArtifact>foo:bar<excludeArtifact> <!-- this one is excluded always  -->
</excludeArtifacts>

? Or do I miss the whole point?

[william-ferguson-au]

In almost all cases skipDependnecies==false.
When it does == true then you will also need special handling for any AAR
dependencies (as they typically contain jars) otherwise the AAR classes
will not be available on the classpath or DEX path.

So I'm starting to think that this might be more work than is worthwhile.
ᐧ

On Wed, May 20, 2015 at 4:10 PM, Lyubomyr Shaydariv <
[email protected]> wrote:

@william-ferguson-au https://github.com/william-ferguson-au
As far as I understand, you mean the ApkMojo only? If so, perhaps does it
mean that is helpful in cases like mine, but there
also should be more precise configuration options letting the dependency
artifacts to be excluded or included individually? Let's say, something
like:

true

foo:bar

and

foo:bar

? Or do I miss the whole point?

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[lyubomyr-shaydariv]

@william-ferguson-au
Yes, I see your point now better, and I see the weakness of simple skipDependencies now for most real life cases, not just mine. I would really like to add more convenient inclusion/exclusion strategies to meet your point better (let's say, inclusion and exclusion by type and by artifacts IDs).

[william-ferguson-au]

The only way it could work would be to specify a list of dependencies to
explicitly excluded from the DEX and compile?) path.

Not sure how the others feel about this direction though.

ᐧ

On Wed, May 20, 2015 at 5:11 PM, Lyubomyr Shaydariv <
[email protected]> wrote:

@william-ferguson-au https://github.com/william-ferguson-au
Yes, I see your point now better, and I see the weakness of simple
skipDependencies now for most real life cases, not just mine. I would
really like to add more convenient inclusion/exclusion strategies to meet
your point better (let's say, inclusion by type and by artifacts).

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[mosabua]

I feel like we should just wait until the Android SDK and so on support Java 8 bytecode ...whatever we do before them will result in a throw away solution and it is potentially a lot of work. I would rather spend our efforts on other things.

[lyubomyr-shaydariv]

I've just pushed the <artifactSet> (something inspired by what I've found the maven-shade-plugin) and <artifactTypeSet> that can be used to specify the inclusions and exclusions more precisely.

They both support <includes> and <excludes> with inner <include> and <exclude> respectively where blank values are just ignored. <artifactSet> allows to include or exclude artifact groups and artifacts by group ID, artifact ID and version (artifact ID and version are optional) strings separated with :. <artifactSet> can override <artifactTypeSet> and <include> has higher priority than <exclude>.

Something like this:

<plugin>
    <groupId>com.simpligility.maven.plugins</groupId>
    <artifactId>android-maven-plugin</artifactId>
    <version>4.2.2-SNAPSHOT</version>
    <extensions>true</extensions>
    <configuration>
        <androidManifestFile>${project.basedir}/src/main/android/AndroidManifest.xml</androidManifestFile>
        <assetsDirectory>${project.basedir}/src/main/android/assets</assetsDirectory>
        <resourceDirectory>${project.basedir}/src/main/android/res</resourceDirectory>
        <sdk>
            <platform>19</platform>
        </sdk>
        <undeployBeforeDeploy>true</undeployBeforeDeploy>
        <proguard>
            <skip>true</skip>
            <config>${project.basedir}/proguard.conf</config>
        </proguard>
        <!--<skipDependencies>true</skipDependencies>-->
        <!--<artifactTypeSet>-->
        <!--<excludes>-->
        <!--<exclude>jar</exclude>-->
        <!--</excludes>-->
        <!--</artifactTypeSet>-->
        <artifactSet>
            <excludes>
                <exclude>foo-group-id:foo-artifact-id</exclude> <!-- any version -->
                <exclude>bar-group-id</exclude> <!-- excludes the whole group -->
            </excludes>
        </artifactSet>
    </configuration>
</plugin>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.10</version>
    <executions>
        <execution>
            <phase>process-classes</phase>
            <goals>
                <goal>unpack-dependencies</goal>
            </goals>
            <configuration>
                <includeScope>runtime</includeScope>
                <includeGroupIds>foo-group-id,bar-group-id</includeGroupIds>
                <outputDirectory>${project.build.directory}/classes</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

<plugin>
    <groupId>net.orfjackal.retrolambda</groupId>
    <artifactId>retrolambda-maven-plugin</artifactId>
    <version>1.8.1</version>
    <executions>
        <execution>
            <phase>process-classes</phase>
            <goals>
                <goal>process-main</goal>
                <goal>process-test</goal>
            </goals>
        </execution>
    </executions>
</plugin>

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
    </configuration>
</plugin>

The result of my initial investigation for the retrolambda-maven-plugin issue is now much wider than I expected it to be, because these commits allow to specify what should be passed to the dx tool, and the Java 8 dependencies support seems to be a special case and you can instrument your dependencies however you like by combining various Maven plugins, not just for Retrolambda. At least I hope so...

This is actually my first Maven-related development attempt and I could probably violate Maven plugin development rules and practices. For example, I'm still not sure whether InclusionExclusionResolver is not re-inventing the wheel (but I believe it is), or whether <artifactTypeSet> is convenient enough, etc.

[lyubomyr-shaydariv]

Guys, any chances this pull request to be merged to the master? I hope this contribution is helpful for someone else too, not just for the ones and me who want to use Java 8 dependencies.

[william-ferguson-au]

I think we are unsure of the worth of adding in a deviation like this. The
complexities it looks like it will add could end up being a royal PITA.
ᐧ

On Mon, May 25, 2015 at 1:47 AM, Lyubomyr Shaydariv <
[email protected]> wrote:

Guys, any chances this pull request to be merged to the master? I hope
this contribution is helpful for someone else too, not just for the ones
and me who want to use Java 8 dependencies.

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[lyubomyr-shaydariv]

Even from the perspective that dx supports any number of input JARs? It would mean that if I had an Ant build, then I could simply invoke the <exec> task to run dx and pass any number of precise arguments to it. Really bad luck then...

[william-ferguson-au]

Hi Lyubomyr, I don't understand your comment.
ᐧ

On Mon, May 25, 2015 at 9:07 AM, Lyubomyr Shaydariv <
[email protected]> wrote:

Even from the perspective that dx supports any number of input JARs? It
would mean that if I had an Ant build, then I could simply invoke the
task to run dx and pass any number of precise arguments to it.
Really bad luck then...

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[lyubomyr-shaydariv]

Sorry for my English grammar/vocabulary (not a native speaker). I mean that I cannot control arguments passed to the dx tool with android-maven-plugin in full. To get direct access to the dx tool, I probably should use maven-exec-plugin that allows to specify any set of arguments passed to an executable which I think is a sort of a dirty way, and I try to avoid using maven-exec-plugin as much as possible. (I addressed Ant in my previous comment, because I often used exec tasks before I moved to Maven -- that might be a confusion too, sorry)

[william-ferguson-au]

From my understanding of the problem you are really looking for a way to
specify which dependencies should NOT be supplied to dex. If that's the
totality then it doesn't seem to bad.

There would only be a small number of use cases requiring this
functionality, but I don't see why we shouldn't support it.

AndroidMaven committers - thoughts?
ᐧ

On Mon, May 25, 2015 at 9:36 AM, Lyubomyr Shaydariv <
[email protected]> wrote:

Sorry for my English grammar/vocabulary (not a native speaker). I mean
that I cannot control arguments passed to the dx tool with
android-maven-plugin in full. To get direct access to the dx tool, I
probably should use maven-exec-plugin that allows to specify any set of
arguments passed to an executable which I think is a sort of a dirty way,
and I try to avoid using maven-exec-plugin as much as possible. (I
addressed Ant in my previous comment, because I often used exec tasks
before I moved to Maven -- that might be a confusion too, sorry)

—
Reply to this email directly or view it on GitHub
#632 (comment)
.

[mosabua]

I think we can probably pull it in but I have to look at it in a bit more detail. Also having some documentation like in this issue in a separate page would be useful.

[mosabua]

I finally merged this. Ideally you could take some of the docs from this PR and put it into a documentation asciidoc file in src/site/asciidoc and send another PR for it. I will release a new version of the plugin as soon I got through the rest of the open PRs.

[lyubomyr-shaydariv]

Thanks for pulling it. My pom.xml won't contain <version>4.3.1-SNAPSHOT-ab2e6706b64dd31a33ece3de77f42c2b2e8606d7</version> anymore. :) Please refer #696.