Bundle everything using "chroot" and "unshare"

[lvml]

Hi,

I wanted to give AppImage a try to create a "RawTherapee 5.1" package - which due to its dependencies is not available for even some recent distributions (like RHEL / CentOS 7.x).

I followed the "Documentation" link on https://appimage.org/, which led me to https://github.com/probonopd/AppImageKit/blob/master/README.md

I read there about LibcWrapGenerator, which I intended to try since it's easy to compile RawTherapee 5.1 on Arch, but it is a terrible pain to try this on CentOS.

Then I noticed that LibcWrapGenerator is not included (or used or mentioned) in the source code that is referred to by the "Source" link on the https://appimage.org/ page - which links to https://github.com/AppImage/AppImageKit instead of the other branch linked above, and both seem to contain pretty different content, with no clear explanation whether one or the other is outdated, and what happened to LibcWrapGenerator in the more recently changed of these two branches.

I also noticed that the "build.sh" script included with AppImageKit does create binaries under build/ - but there is no word on whether those should be installed somehow (if only by copying them to /usr/local/bin/ or something).

Can you clarify this confusion?

[TheAssassin]

You don't actually have to use CentOS, just use something sufficiently old. Debian (old)stable or Ubuntu 14.04 or openSUSE Leap 42.2 will work fine for that purpose, too. But Arch is a bad idea, please see below for details.

I am not really aware of LibcWrapGenerator and how it works, but it sounds hackish, so I'd avoid that.

To make your life easier and since you are obviously not necessarily interested in building AppImages for the latest (and probably greatest) development code, I'd recommend you to check out our latest innovation.

You probably haven't heard of that, but Open Build Service (including the free to use openSUSE build service) can build AppImages now. And guess what: The easiest way is to repackage existing RPMs.

openSUSE chairman Richard Brown and AppImage creator @probonopd explain why this is a good thing and why it greatly simplifies the creation of AppImages in these videos:

• https://media.ccc.de/v/1320-opensuse-jurrassic-park
• https://www.youtube.com/watch?v=8r1OwX-dNFo

Our wiki contains a really good guide to that: https://github.com/AppImage/AppImageKit/wiki/Using-Open-Build-Service

OBS contains a pretty up to date RPM in the graphics repository.

Now all you have to do is set up a project on OBS, add the graphics repository to your project on OBS, add the YML file describing the build and let OBS do the rest!

Features:

• automatic rebuilds of the AppImage whenever a bundled library or RawTherapee are updated
• automatic support for AppImageUpdate
• write the YML once, never touch it again

If you are interested in doing it that way, it would be a great help for our project, since we really need testers for the instructions and OBS, and I'd highly appreciate if you went that way. Only if that is what you want to do, of course.

If you need help with the details (you most likely will, as the OBS documentation isn't that complete, and we've probably faced the problems already), feel free to join us in #AppImage on Freenode at any time. We'll help you to make the AppImage recipe.

If you insist on building from source, OBS can do that too. If you don't wish to use OBS, that's fine too.

A few more comments on your initial problems:

• Please, do NOT build AppImages on Arch Linux. Even if you could technically make them with Arch and that wrapper thingy, it'd most likely be really hard for people to rebuild your AppImage. Arch, due to its rolling release character and the mostly questionable "package management" isn't at all suitable as a build system for anything.

• The README might be outdated. Please open another issue that describes what doesn't work, and we'll try to fix that.

• The build.sh script is intended to build AppImageTool of which an AppImage is created. Of course, we ship our build tools as AppImages, too. You can just go to the releases page, and download the latest copy of the AppImage. You do not have to install these AppImages. Following our AppImage spirit, just download them, make them executable and run them.

• Please note that we also provide linuxdeployqt, which aims to simplify the creation of AppImages. Don't interpret that name wrong, it's just written in Qt, but works well for any software.

I hope that helps you and gives you a good overview of the purpose of this repository.

[probonopd]

Hi @lvml, welcome to AppImage.

I also noticed that the "build.sh" script included with AppImageKit does create binaries under build/ - but there is no word on whether those should be installed somehow (if only by copying them to /usr/local/bin/ or something).

We generally recommend using our prebuilt binaries, https://github.com/AppImage/AppImageKit#building

If you want to build yourself, it should be sufficient to add the build/ directory to your $PATH.

[probonopd]

Please, do NOT build AppImages on Arch Linux. Even if you could technically make them with Arch and that wrapper thingy, it'd most likely be really hard for people to rebuild your AppImage. Arch, due to its rolling release character and the mostly questionable "package management" isn't at all suitable as a build system for anything.

And even more importantly, building on a Rolling Release distribution would mean that your AppImage would only run on the very latest target systems, as is explained in https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages#binaries-compiled-on-old-enough-base-system

[lvml]

Hmm... thanks for your explanations, but they seem to address a different use case than the one I am currently contemplating about:

I own several computers, one of which runs CentOS 7, and I would like to run the same up-to-date version of RawTherapee on that one as on the other, Arch-Linux-running computers.
Compiling RawTherapee from its sources on Arch is a matter of minutes, but on CentOS 7, it's a bottomless pit of unresolved dependencies, if only because of the gtk3 absence.

My idea was to create an AppImage on Arch, for the sole purpose of running it on the CentOS 7 machine. Of course I am prepared to copy vast amounts of binaries, libraries and data files into that AppImage to make this work, I don't mind its size and would even be fine with putting ld-linux.so.* and libc with all its additional shared objects and data files in there, if required.

But compiling RawTherapee 5.1 on some older operating system like CentOS in order to create an AppImage from this completely defeats the purpose of the whole exercise: If I could make this work with reasonable effort, I could compile RawTherapee 5.1 on the target system in the first place and be done (no need for an AppImage, then).

Regarding:

building on a Rolling Release distribution would mean that your AppImage would only run on the very latest target systems

Maybe my understanding of AppImage was wrong, but I thought the very purpose of AppImage is to put everything required by an application (south of the kernel ABI) into a binary such that it can be run on any Linux distribution?
(By the way: That is what we do at work: Compile software on a very recent Linux system, and linking statically so everything is included and can run even on older systems.)

If "building on a recent system" means "cannot run on an older system", then AppImage may just not be meant for the purpose I'm after...

[probonopd]

@lvml you can decide what you put inside an AppImage, so you could do what you suggest. Please see AppImageCommunity/pkg2appimage#84

[probonopd]

We don't generally recommend to bundle everything though, since this could be percieved as bloated. Instead, we do not bundle libraries that can reasonably be expected to be part of every target system (e.g., glibc and friends). Which means that we need to compile on a system no newer than the oldest target system.

[lvml]

Thanks for the link to AppImageCommunity/pkg2appimage#84, will have a look at the options discussed there.

(BTW: Did anyone ever try to use https://github.com/rpodgorny/unionfs-fuse to user-mount the content of an AppImage as an "overlay" to the original root file-system? This would seem like an obvious idea to me for a "bundles everything" kind of AppImage, solving the issue that one can use absolute paths in the application just normally, even for the dynamic linker.)

Regarding "bloat": I agree that "bundling everything" might be undesirable for those who intend to publish an application of theirs to a large audience. However, for the use case of "just make application X currently running fine on computer Y run also on computer Z", a "bloated" AppImage is the least of my concerns. Should such an image of RawTherapee allocate 1GB of disk space - I woudn't mind, that is still below 1/1000 of the available storage space on the target, and I do not intend to populate the target with thousands of AppImages.

[probonopd]

Yes, https://github.com/rpodgorny/unionfs-fuse is an option. Whoever puts together an AppImage can put it inside, with a script called "AppRun" that sets it all up.

[lvml]

Success! (Albeight not using AppImage, but I wanted to let you know anyway since the information you provided in this thread was very helpful for my inspiration - thank you.)

I managed to run the binaries from an Arch Linux installation under CentOS:

I created a script that uses "strace ... rawtherapee" under Arch to record all the files that rawtherapee requires - including all distribution-provided shared libraries, data files and links.

Then I tar'ed those files, copied them to the CentOS installation, untar'ed them there (into /home/test7/rt_root), and used (as a normal user named "test7"):

unionfs -o dev,relaxed_permissions /home/test7/rt_root=RW:/=RW /home/test7/union_root

and then I can start (also as a normal user) rawtherapee using:

LD_LIBRARY_PATH=/lib64:/usr/lib64 unshare -r chroot union_root /usr/bin/rawtherapee

This very conveniently solves my use case (without requiring any re-compilation of RawTherapee under some older OS).

If I find the time, I'll try to wrap up my scripts into a piece of software that then could be used for just any binary, provided the kernel is "new enough" on the target system.

And of course it may also be a good idea to use something like https://github.com/solidsnack/arx to make the archive self-extracting / runnable.

[probonopd]

Well, you can throw this into an directory, write a bash script called AppRun that does the unionfs magic and launches the application, run appimagetool on it and have a working AppImage...

In fact, we'd be very interested in this.

[lvml]

@probonopd: Yes, that could make is unnecessary to unpack the archive on the target, just mounting the squashfs image instead. Will try that.

(Of course this will also require me to create statically linked versions of "unionfs" and "unshare" - but those should not be hard to create.)

BTW: The rt_root directory I assembled isn't that big, despite containing even libc, libstdc++ and their related data files - 210MB unpacked, 50MB tar.xz'ed - that space is really worth not having the hassle of trying to compile that software on the old distro ;-)

[probonopd]

unionfs can go into the AppImage as well, so that it doesn't have to be there on the target system...

[lvml]

What works: I have prepared static versions of chroot, unshare and unionfs, plus an "AppRun" script that mounts the unionfs and runs the application fine on CentOS.

What does not work: After a longer-than-necessary-time to guess the right format of the ".desktop" file (seriously, this needs better documentation), I did get a ready AppImage file, but running that on CentOS fails:

/tmp/RawTherapee5.1-x86_64.AppImage 
fusermount: bad mount point /tmp/.mount_BXxCZO/union_root: Permission denied

It seems like mounting a unionfs on a mount-point inside the (also user-mounted) squashfs does not work.

[probonopd]

Thanks @lvml, can you upload it somewhere, e.g., on http://transfer.sh?

[lvml]

Ok, worked around the mount directory issue by using $(mktemp -d) as a mount directory.

Only not-nice thing remaining: I have to use "killall unionfs_static" to unmount the unionfs as a user, as I could not find a way to find out the PID of the unionfs process that I spawned when mounting - this would be bad if two such AppImages were used at the same time.
Any idea how to find the PID of such a fuse process?

I'm uploading the resulting RawTherapee5.1-x86_64.AppImage as I write this - my connection is not the fastest, so this might take a while.

(BTW: It's getting late here and I'll need to go to sleep soon)

[lvml]

Ok, so here's my https://transfer.sh/jKzgE/RawTherapee5.1-x86_64.AppImage in case anyone wants to try it on other distributions.

And remember: It was done without compiling RawTherapee at all, just using the Arch Linux supplied binaries, so no warranties from my side ;-)

[probonopd]

me@host:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.10
DISTRIB_CODENAME=quantal
DISTRIB_DESCRIPTION="Ubuntu 12.10"

me@host:~$ '/home/me/Downloads/RawTherapee5.1-x86_64.AppImage' 
fuse: warning: library too old, some operations may not work
unshare_static: unshare failed: Invalid argument

# The offending command is:

+ squashfs-root/appimage_support_binaries/unshare_static -r squashfs-root/appimage_support_binaries/chroot_static /tmp/tmp.FPVrHfMyoo /usr/bin/rawtherapee
unshare_static: unshare failed: Invalid argument

[probonopd]

me@host:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.1 LTS"

# Many such errors:

Error creating proxy: Exhausted all available authentication mechanisms (tried: EXTERNAL, DBUS_COOKIE_SHA1, ANONYMOUS) (available: EXTERNAL, DBUS_COOKIE_SHA1, ANONYMOUS) (g-io-error-quark, 0)
Error creating proxy: Exhausted all available authentication mechanisms (tried: EXTERNAL, DBUS_COOKIE_SHA1, ANONYMOUS) (available: EXTERNAL, DBUS_COOKIE_SHA1, ANONYMOUS) (g-io-error-quark, 0)
Error creating proxy: Exhausted all available authentication mechanisms (tried: EXTERNAL, DBUS_COOKIE_SHA1, ANONYMOUS) (available: EXTERNAL, DBUS_COOKIE_SHA1, ANONYMOUS) (g-io-error-quark, 0)

But in the end it runs!

[probonopd]

Any idea how to find the PID of such a fuse process?

You can use & and $! like so:

$SUPPATH/unionfs_static -o dev,relaxed_permissions "$THISDIR=RW:/=RW" "$UNIONPATH" &
UNIONFS_PID=$!
...
kill $UNIONFS_PID

But I think the really clean solution would be to combine all of

me@host:~$ ls -lh appimage_support_binaries/
total 3.4M
-rwxr-xr-x 1 me me 830K Jun 12 08:41 chroot_static
-rwxr-xr-x 1 me me 786K Jun 12 08:41 fusermount_static
-rwxr-xr-x 1 me me 1.1M Jun 12 08:41 unionfs_static
-rwxr-xr-x 1 me me 750K Jun 12 08:41 unshare_static

into one C executable (if licenses are compatible, and if these are known not to be available on otherwise supported distributions; to be verified) to make it more robust and avoid shell scripting.

At the very least, bash atexit() should be used so that the fuse process also gets killed if the application exits prematurely (is killed). Like so:

$SUPPATH/unionfs_static -o dev,relaxed_permissions "$THISDIR=RW:/=RW" "$UNIONPATH" &
UNIONFS_PID=$!

trap atexit EXIT

atexit()
{
  set +e
  kill $UNIONFS_PID
}

[probonopd]

For size comparisons, it would be useful to see such an AppImage for Leafpad.
https://www.archlinux.org/packages/extra/x86_64/leafpad/

[lvml]

Regarding "unshare_static: unshare failed: Invalid argument": That happenes if CONFIG_USER_NS is not set in the linux kernel .config - without this, there is no way for a non-privileged user to utilize chroot.

Regarding UNIONFS_PID=$!: No, that won't work, unionfs forks off a persistent process, the PID you can get via $! is not the PID of that process.

[lvml]

For size comparisons, it would be useful to see such an AppImage for Leafpad.

Well, that's easy to answer (since creating such images is really easy): https://transfer.sh/rO4gV/leafpad-x86_64.AppImage is 29MB in size. The largest contributor, by the way, is /usr/lib/libicudata.so.59.1

And a "hugin" image is ~80MB in size.

[lvml]

BTW: I hope this annoying bug will be fixed, as some software really likes to read/write/mmap /dev/null or /dev/zero.

[lvml]

I brushed up and uploaded my software (named "makeaoi") for others to use, please try it:

https://github.com/lvml/makeaoi

[lvml]

"makeaoi" now also supports tunneling (some, currently mostly serial-device related) ioctl()s and has support for poll()/select() now.

Here is an example AppImage making use of those features: https://transfer.sh/2tLny/subsurface-4.6.4-2-x86_64.AppImage

[probonopd]

tunneling (some, currently mostly serial-device related) ioctl()s and has support for poll()/select()

Forgive my ignorance, but what is this needed for, or asked more precisely, what symptoms/error messages should I watch out for that would indicate that this is needed?

[lvml]

On 07/31/2017 08:05 AM, probonopd wrote: tunneling (some, currently mostly serial-device related) ioctl()s and has support for poll()/select() Forgive my ignorance, but what is this needed for, or asked more precisely, what symptoms/error messages should I watch out for that would indicate that this is needed?

If an application (like "subsurface") attempts to communicate via a device (like /dev/ttyUSB0), it often does ioctl()s on the opened file and also uses poll() or select() to wait for writeability or readability of the device. When attempting to access a device file this way that is exposed via a mounted unionfs-fuse (without my changes and "-o fake_devices"), ioctl() calls will fail and poll() calls will return immediately, so the application will barf on failed communication.

[probonopd]

Another option: https://intoli.com/blog/exodus-2/

[lvml]

Thanks for pointing to Exodus as another alternative. However, its list of limitations includes two prominent ones that rule out its use for my use cases: It does not bundle glibc, and it does not bundle script-interpreters.
It's a pity that unionfs-fuse does not seem to be actively maintained anymore, because some bugs in it prevent a number of use-cases that could be covered by "makeaoi".

BTW: I noticed the RawTherapee makers now offer an AppImage for download - and of course I tried it. Unluckily, it did not work on my CentOS 7 target system, due to missing "wayland" libraries.

[TheAssassin]

@lvml you should open an issue in their repository, then.