deblobbify, replacing the system folder with several other components

[robertkirkman]

Potential WIP/PoC solution to:

• termux-docker images should be properly labeled with their native arch #35
• The whole system directory may be removed from the repo... #60
• Hostnames fail to resolve #63
• Missing TERM environment variable #65

Summary of changes:

• Deletes the entire system folder and replaces it with a script generate.sh that can regenerate the entire system folder.
• The root filesystem, including system folder, is regenerated using purely a Termux bootstrap .zip, and .deb files that can be built using the termux-packages build and CD system, then retrieved from the termux.dev package host.
• Adds an environment variable $TERM that is set to a value of "xterm" for the clear command to have convenient compatibility with most terminal emulators, and this has also been tested with Kitty.
• Simplifies the instances of the string "com.termux" into a single instance for slightly more convenient use with custom bootstraps.
• Fixes the existing docker.io/termux/termux-docker:aarch64 image not working on aarch64 devices that happen to have an esoteric "Cortex-A34"-type or similar processor due to the elimination of this 32-bit ARM binary blob, which Cortex-A34-type processors do not have hardware support for executing, unlike "Cortex-A72"-like processors.
• Replaces Busybox with a real build of Toybox based on the source code of AOSP 9.0.0 that has been patched to meet the requirements of termux-docker, including providing su to log in as the system user within entrypoint.sh.
• Uses mksh and bash alone to implement sh, instead of Busybox's sh.
• Uses a build of iputils based on the source code of AOSP 9.0.0 to implement /system/bin/ping and /system/bin/ping6.
• Uses the preexisting termux-packages root/dnsmasq to implement dnsmasq, which works after some adjustments to its invocations.
• Adds a third method of working around the personality() system call permissions issue, a custom build of Docker, which is not required on all ARM devices, but which might be required sometimes, and is required on my armv7l device that the --privileged and --security-opt arguments both failed to work on.
• Replaces the use of QEMU in GitHub Actions with the new official public GitHub ARM runner.
• Enables updating the termux packages within the container before publishing the ARM images.
• Deletes /system/etc/ld.config.28.txt without replacing it with anything, since it does not appear to be necessary for anything in any test so far, but it could be easily re-added if it turns out that there is something that needs it.

Tested both locally and in GitHub Actions:

• aarch64: working
• x86_64: working
• arm: working, but in some (not all) cases, the custom build of Docker with modified personality() system call permissions is required. It is unknown how to predict whether a device will require that or not, but it is suspected it might have something to do with the host kernel, which is a factor whose properties it is understood tend to vary dramatically between different ARM devices.
• i686: working

The "temporary" things like the temporary-package-storage folder and the blobs inside are only necessary for testing while this is not merged:

• bionic-host: rename to aosp-libs, bump to 9.0.0-r76, add aosp-utils subpackage, revbump revdeps termux-packages#22906

The large amount of changes to the internal components of termux-docker could introduce unpredictable bugs or unexpected behavior that is not present with the current official termux-docker. For that reason, I hope to learn all of the current intended use cases that termux-docker is officially supposed to be used for, so that I can test my implementation with them and add additional software or fixes if needed.

Known Unique Limitations:

• For some reason, the -c argument of toybox-su is currently non-functional. For the current featureset of termux-docker, both before and after this PR, that does not seem to be a major priority because termux-docker currently only supports the root user via the entrypoint_root.sh and not via su or tsu from within the default entrypoint.sh, i.e. most termux-packages root-packages cannot currently be conveniently tested using termux-docker anyway. If "proper simulation of a rooted device", so to speak, is desirable in the future, to facilitate more convenient testing of root-packages, that could be investigated and handled by a different issue+PR at a later date.

TODO:

• remove the temporary-package-storage folder and all references to it (blocked by the bionic-host PR)

[twaik]

modifying the termux-packages bionic-host package is not acceptable

Nobody said it is not acceptable. I think nobody tried it.
I created the package as a solution for running android executables during regular building with toolchain to avoid using hostbuild step, but for some reason qemu does not want to run fine with aarch64 executables on amd64 host, at least in some cases.
Also it fits fine for using in termux-docker but it was not modified to use bionic-host package.

[robertkirkman]

for some reason qemu does not want to run fine with aarch64 executables on amd64 host, at least in some cases

I am not sure exactly what exact scenario you are talking about here, or what error it is you were seeing at the time, if you mean a scenario with a build.sh involving only the regular cross-compiling toolchain, then what I am about to say is probably not relevant,

But just in case knowing this helps you, there is a way to emulate an ARM processor on an x86 device that is very high in emulation accuracy and behaves almost identically to how real ARM devices behave.

When carefully configured, I have never seen qemu-system-aarch64 fail to execute ARM software on an x86 processor, as long as that same ARM software also works on real ARM devices. If I ever see frustrating errors while trying to use user-mode emulation of any kind, qemu-system-aarch64 is my go-to solution for accuracy to the exact behavior of real ARM devices, when I am in a place with only x86 devices and accessing real ARM devices is inconvenient.

Since there are very few good guides on the subject, and almost all other guides skip mentioning lots of necessary steps, leaving everything unexplained and causing most people to feel like qemu-system-aarch64 is much too tedious to consider using on a regular basis, I created this guide that covers every detail of the setup process in a very straightforward, completely explained, and future-proofed way based on the latest UEFI technology, which is also distro-agnostic to both the host and the guest, making it very clear what changes when the guest and host are mixed and matched.

[twaik]

I will just link this.
termux/termux-packages#18517 (comment)
I know it is not a simple scenario, but GIR-generated binary used for generating some XMLs for some reason hangs and CI is terminated with timeout after 6 hours so it is a no go for us. It was the very first and the last attempt (my attempt) to use cross-compiled binaries for building (to avoid using host_build step).

[robertkirkman]

GIR-generated binary used for generating some XMLs for some reason hangs and CI is terminated with timeout after 6 hours so it is a no go for us

I experimented with some local tests based on qemu-system technology (as opposed to qemu-user technology),

and while it can inherently avoid several of the issues described in that thread, such as the GIR hanging problem, the problems with any other binaries hanging, and the pid_max problem, due to the higher emulation accuracy and environment micromanagement of qemu-system compared to qemu-user,

very unfortunately, it cannot overcome the slowness problem that licy183 explained. Additionally, the higher emulation accuracy of qemu-system comes with a cost that the slowness is even more extreme, causing the build time for multiple things I need to build to go far over the 6 hours limit (my builds were successful locally, but took 16 hours). Even though that technically works for me locally, that makes it not viable for anything in GitHub Actions without also using the 6 hour limit bypass incremental build workaround technique every time.

I think the only efficient possible future plan on this topic, is to make use of termux-docker, or licy183's tur-avd, or another minimal android container, combined with both the new GitHub ARM runners and with the termux-packages clang package, to build some packages with cross compilation disabled and emulation also disabled.

I am already testing the GitHub ARM runner in this PR and it works for example here,
but since TUR has already been building packages in this same codepath using self-hosted ARM runners instead of cloud ARM runners, it is possible that maybe licy183 is already about to or in the process of adding the cloud ARM runners to TUR, which could serve as an example of how to proceed.
They had tested a PR using it a few months ago, but I did not see any other tests of it yet.
If they do not end up adding it soon, I could attempt my own way of adding support for it to the termux-packages repo, but first I need to finish the changes to termux-docker.

I cannot complete the changes to termux-docker without first removing blockers, like fixing the builds of pypy and pypy3 so that the dependency can be merged. Because of that, I think I am currently forced to first use the only workaround possible to force them to build in a single PR with the updated aosp-libs package, the pid_max change, then I could attempt to migrate the ARM builds of pypy and pypy3 to an actual ARM runner some time afterward, after creating it. The pid_max change would remain required for running any 32-bit termux-docker-based workflows anyway, so it could be moved to the termux-docker-based workflow instead at the time that is finished.

I have tested locally that pypy would build and work using termux-packages clang inside a complete termux-docker ARM system, it is just that there is no workflow set up for that yet. I would prefer waiting to try introducing termux-docker to such a workflow until after it has had the blobs removed.