Build Android tools from source (adb, SDK, Android Studio, etc.)

[primeos]

We should build as much Android tools as possible from source. The most relevant tool is probably the adb (Android Debug Bridge) to communicate with Android phones.

Reason

The official builds from Google are distributed under an unfree license (see #53403).

Difficulty

Probably high.

Resources

• References:
  • Debian efforts: AndroidTools
  • Arch package: android-tools
• adb (and other Android tools)
  • Requires a custom build system: Soong
    • Not in Nixpkgs yet -> wee need a package and an abstraction for building packages with Soong first.
    • Official documentation: https://source.android.com/setup/build
  • Source-code: https://android.googlesource.com/platform/system/adb/+/refs/heads/master
• Android Studio
  • Official documentation: https://android.googlesource.com/platform/tools/base/+/studio-master-dev/source.md
  • Tried it once (all information IIRC, without any guarantees), the build required a gitRepo checkout of ~40GB. Then there are multiple steps involved (without much documentation), but the the build itself was working. The main problem problem is to put all results back together (IntelliJ IDEA, the Android "plugin", and the Android Gradle plugin). And we also don't build IntelliJ IDEA from source so that might be a problem as well.

[matthewbauer]

Related to:

• Build Android tools from source (adb, SDK, Android Studio, etc.) #75603
• Nixos android sdk license #23910

[Atemu]

One more reason to do this: Nixpkgs' ADB is only available on x86

[tgunnoe]

It is currently impossible to accept the android license when the sdk sits in the nix store, and is installed through nixpkgs. So this is probably needed.

#76420 (comment)

[danielfullmer]

First, note there is an android rebuilds project which aims to provide license unencumbered builds of the android SDK.

Also, I've been working on a project (robotnix) to build Android ROMs using nix.

As an experiment, I looked into whether it could be leveraged to build the android SDK, with some success: https://github.com/danielfullmer/robotnix/blob/master/sdk/default.nix
To build:

nix-build -E 'import ((builtins.fetchTarball "https://github.com/danielfullmer/robotnix/archive/master.tar.gz") + "/sdk")'

Which produces the following files under ./result:

android-sdk_eng.10.0.0_r33_linux-x86.zip
repository.xml
repo-sys-img.xml
sdk-repo-linux-build-tools-eng.10.0.0_r33.zip
sdk-repo-linux-docs-eng.10.0.0_r33.zip
sdk-repo-linux-platforms-eng.10.0.0_r33.zip
sdk-repo-linux-platform-tools-eng.10.0.0_r33.zip
sdk-repo-linux-samples-eng.10.0.0_r33.zip
sdk-repo-linux-sources-eng.10.0.0_r33.zip
sdk-repo-linux-system-images-eng.10.0.0_r33.zip

There are a few caveats:

1. The source is huge. >40GB is required just to check out the AOSP source. To avoid extraneous copying, I use namespaces to bind-mount the source into its proper location during the build.
2. It takes a very long time to build. It takes approximately 4 hours on my i3770. I have since upgraded to a 32-core threadripper which can do this in about 25 minutes.
3. It relies on Google's prebuilt toolchain included in the AOSP checkout. This only has support for linux-x86 and darwin-x86. We still have to autoPatchelf the tools which are intended to be run on the host.
4. It relies on the module system I'm using for robotnix, which probably doesn't belong in nixpkgs. This could (with a little work) be removed to produce some relatively simple nix code for building the sdk.

Instead of building the entire SDK, I also looked into building only adb and fastboot here:
https://github.com/danielfullmer/robotnix/blob/master/sdk/adb.nix
This is much faster, but unfortunately still requires checking out the entire AOSP source. I spent a little time trying to exclude various subprojects, but soong (android's new build system) would complain about missing references.

As far as inclusion in nixpkgs: I think an alternative package for adb/fastboot which doesn't require accepting the SDK license would be most valuable for users. However, I'm not sure such a package would be accepted into nixpkgs if it requires checking out all of AOSP.

[xaverdh]

Instead of building the entire SDK, I also looked into building only adb and fastboot here:
https://github.com/danielfullmer/robotnix/blob/master/sdk/adb.nix
This is much faster, but unfortunately still requires checking out the entire AOSP source. I spent a little time trying to exclude various subprojects, but soong (android's new build system) would complain about missing references.

As far as inclusion in nixpkgs: I think an alternative package for adb/fastboot which doesn't require accepting the SDK license would be most valuable for users. However, I'm not sure such a package would be accepted into nixpkgs if it requires checking out all of AOSP.

Maybe somebody with more knowledge of the hydra build servers / infra can tell if that would be a problem.

But is soong actually insisting on unneeded references, or are all of those really needed?
Maybe you could trace the build process and just track all files read during building, then set up a git repository with just these files in it? If it works well, this could be a fully automated process.

[danielfullmer]

@xaverdh So the cause of the soong failure is that it walks the entire android source tree and tries to resolve all the build references in the blueprint files before building anything. However, I recently found that soong has an ALLOW_MISSING_DEPENDENCIES environment variable, which does seem to ignore some missing dependencies. I've tried using this environment variable and adding only the repos based on the error messages I get while building adb/fastboot. Even with this, I still get a number of depends on undefined module errors that refer to modules that seem entirely unrelated to adb/fastboot. It might be possible to resolve these issues with additional work...

[musfay]

If we only want to build tools, there is a friendly build-system for it. https://github.com/nmeum/android-tools
Fedora and Arch already switched to it, openSUSE devs are also thinking about switching to it. The source tarball is about 24mb. Also we can create our tarballs with build-system. I think it's worth a look.

[primeos]

@musfay that's really awesome, thanks a lot for mentioning it! Seems like the Alpine Linux package maintainer created this and Void Linux is also using and contributing to it. I've drafted #124992 for Nixpkgs but I/we need to figure out how to best fetch the vendored Go modules first (then the rest of the build will hopefully succeed).

[samueldr]

#124992 is now merged, closing :)

[malte-v]

This should be reopened. Currently we're building adb and fastboot from source, but the SDK and Android Studio are still proprietary binaries with license agreements attached to them.

[samueldr]

Indeed. Another issue with a broad scope which becomes hard to close.

[malte-v]

Probably relevant: nix-community/robotnix#107

[stale]

I marked this as stale due to inactivity. → More info

[Ericson2314]

Keep alive, we got closer with Android cross, and perhaps robotnix can help.

[hello-smile6]

Is it likely that this will work soon?

[xaverdh]

Well the nixos adb module could already use adb built from source (I opened #136040 for that). Everything else will probably take some time.

[zkpgbig]

If we only want to build tools, there is a friendly build-system for it. https://github.com/nmeum/android-tools
Fedora and Arch already switched to it, openSUSE devs are also thinking about switching to it. The source tarball is about 24mb. Also we can create our tarballs with build-system. I think it's worth a look.
I want to compile build-tools located in build/tools folder in source. make sdk-eng?

[Atemu]

@zkpgbig android-tools is already packaged from source.

[zkpgbig]

@zkpgbig android-tools is already packaged from source.

I just want to build it from source…

[Atemu]

nix-build '<nixpkgs>' -A android-tools