Alpine libs not compatible with Java 8

[giovannicandido]

Trying to install oracle JDK 8 and I get this error when run:

. /opt/jdk/bin/java -version
/bin/ash: /opt/jdk/bin/java: : not found
/bin/ash: /opt/jdk/bin/java: : not found
@@@@@@@?@@@@ |?@@ddp?td??@?@1q?t/lib64/ld-linux-x86-64.so.2GNU GNU?t?)N?ݓ??;r?Z???h nonexistent directory
/bin/ash: /opt/jdk/bin/java: line 1: ELF: not found
/bin/ash: /opt/jdk/bin/java: ?: not found
/bin/ash: /opt/jdk/bin/java: line 3: ?: not found
/bin/ash: /opt/jdk/bin/java: syntax error: unexpected end of file (expecting ")")

The ldd command give the result:

ldd /opt/jdk/bin/java
    /lib64/ld-linux-x86-64.so.2 (0x7f8da09f8000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f8da09f8000)
    libjli.so => /opt/jdk/bin/../lib/amd64/jli/libjli.so (0x7f8da07e0000)
    libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f8da09f8000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f8da09f8000)
Error relocating /opt/jdk/bin/../lib/amd64/jli/libjli.so: __rawmemchr: symbol not found

What could be done?

[andyshinn]

This is more of a musl libc issue. The Oracle Java binaries only run on glibc at the moment. We are looking at possibly rebuilding https://aur.archlinux.org/packages/jd/jdk/PKGBUILD and making it available as a standalone package. I'll keep this issue open for now as we have had requests for Java 8 multiple times already.

[cultureulterior]

Relevant to this is that the java 7 package is slightly broken too- the certstore is broken causing very hard to diagnose errors. The alpine linux irc guys say to copy it in from a debian install

[AntonioMeireles]

(curious) @cultureulterior could you providde more details in your findings plz.

[cultureulterior]

@AntonioMeireles the java certstore apparently is not regenerated from the CA certs on the machine itself, causing the java ssl to fail to trust certificates that should be trusted

[progrium]

I think I may start supporting progrium/busybox again just for Java 8 ... can you guys try that as your base image?

[pires]

+1

[frol]

Can anyone point me to relevant information about these musl libc limitations? I have the similar issue with Anaconda prebuilt packages on Alpine Linux:

# ldd /usr/local/miniconda/pkgs/python-2.7.9-1/bin/python
        /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
        libpython2.7.so.1.0 => /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0 (0x7f9d68a63000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
        libutil.so.1 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
        libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f9d68e45000)
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __finite: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __rawmemchr: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __isinff: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: tmpnam_r: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __isnan: symbol not found
Error relocating /usr/local/miniconda/pkgs/python-2.7.9-1/bin/../lib/libpython2.7.so.1.0: __isinf: symbol not found

I do understand that such prebuilt packages are evil, but there is not much I can do in this case except moving to another image for now. :(

However, I would like to understand limits of musl libc better.

[pires]

/cc @FlavioF @albertocsm

[denvazh]

@cultureulterior not directly related to this issue, but it is possible to manually import all ca certificates to java trusted keystore, as I did it here

[andyshinn]

We have a new glibc package if anyone would like to test this out. You can grab the package at https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk and install with apk add --allow-untrusted glibc-2.21-r2.apk. This should allow Oracle Java 8 and other glibc x86_64 native binaries to run on Alpine.

[frol]

@andyshinn Awesome! It works for me with Miniconda, though I had to bring libz.so from my ubuntu to /usr/lib/libz.so.1 as Alpine Linux has libz.so compiled against musl...

[frol]

@andyshinn
UPD: I didn't need to bring libz.so from ubuntu, but needed to create symlinks:
/usr/lib/libc.musl-x86_64.so.1 pointing at /lib/libc.musl-x86_64.so.1
/usr/lib/libz.so.1 pointing at /lib/libz.so.1

Note: For some reason, LD_LIBRARY_PATH doesn't work in this case for me.

[andyshinn]

Are you able to share a Dockerfile or repository with this? I'd like to see how this is working to better support these edge cases.

[frol]

@andyshinn Sure!

[frol]

@andyshinn

FROM alpine

RUN apk update && \
    apk add bash wget

WORKDIR /tmp

RUN wget --no-check-certificate "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk" && \
    apk add --allow-untrusted glibc-2.21-r2.apk

RUN wget "http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh"

# This will fail with "ImportError: libz.so.1: cannot open shared object file: No such file or directory"
#RUN bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda

# This will succeed
RUN ln -s /lib/libc.musl-x86_64.so.1 /usr/lib/libc.musl-x86_64.so.1 && \
    ln -s /lib/libz.so.1 /usr/lib/libz.so.1
RUN bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda

[andyshinn]

I removed the post-install ldconfig but it might actually cover this case. Can you try adding package https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-bin-2.21-r2.apk in addition to the glibc one and then run /usr/glibc/usr/bin/ldconfig /lib /usr/glibc/usr/lib before installing and see if it works?

[frol]

@andyshinn Now it works without symlinking! Great!

Just for the history (it works like a charm):

FROM alpine

RUN apk add --update bash wget ca-certificates

WORKDIR /tmp

RUN wget "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk" && \
    apk add --allow-untrusted glibc-2.21-r2.apk && \
    wget "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-bin-2.21-r2.apk" && \
    apk add --allow-untrusted glibc-bin-2.21-r2.apk && \
    /usr/glibc/usr/bin/ldconfig /lib /usr/glibc/usr/lib

# Hotfix for glibc hack that fixes the order of DNS resolving (i.e. check /etc/hosts first and then lookup DNS-servers).
# To fix this we just create /etc/nsswitch.conf and add the following line:
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf

RUN wget "http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh"

RUN bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda

In real world, I suggest you write something like:

FROM alpine

# Here we use several hacks collected from https://github.com/gliderlabs/docker-alpine/issues/11:
# 1. install GLibc (which is not the cleanest solution at all) 
# 2. hotfix /etc/nsswitch.conf, which is apperently required by glibc and is not used in Alpine Linux

RUN apk add --update bash wget ca-certificates && \
    wget "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk" \
         "https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-bin-2.21-r2.apk" && \
    apk add --allow-untrusted glibc-2.21-r2.apk glibc-bin-2.21-r2.apk && \
    /usr/glibc/usr/bin/ldconfig /lib /usr/glibc/usr/lib && \
    echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
    wget "http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh" && \
    bash ./Miniconda-latest-Linux-x86_64.sh -b -p /usr/local/miniconda && \
    rm ./Miniconda-latest-Linux-x86_64.sh && \
    apk del bash wget ca-certificates && \
    rm /var/cache/apk/*

This way you avoid packaging unnecessary packages (bash, wget), giant installer (rm miniconda.sh), and outdated apk repository lists.

[andyshinn]

A couple of things I want to point out:

1. Your Python is not part glibc and part musl. Now, I'm a progressive minded person and don't have a problem with this. But Python might... A dual C library stack like this probably isn't all too common so we don't know what the pitfalls are (if any).
2. When you install pip and then start building more Python extensions (ones requiring build tools such as build-base) those will be linked against musl (be aware of DNS implications this has).
3. If you are not building additional extensions then we can probably pull in the libz.so from Arch to break the dependency on musl. Soon, i'll have an example of what a package like this looks like

[frol]

@andyshinn I don't really use Miniconda it was just a project that I played with. My concern was mostly about the limits of musl libc than about running Miniconda on Alpine Linux. Anyway, thank you for your effort to get things easier in Alpine. I believe, Java people will appreciate this a lot.

[andyshinn]

Fair enough, it is something that applies to anyone trying to use glibc on Alpine so just wanted to get it out there. We are definitely more targeting the edge cases of one-off glibc binaries (think Consul) or Java 8 applications that might have all dependencies bundled in a war file already. But it is helpful to see where the limits are, so thank you for that!

[frol]

@andyshinn Another example, which may be considered harmful, but nevertheless it made the shortcut possible, is my frolvlad/alpine-mono image, where I use mono package from Arch Linux.

[frol]

Here is the Apline OracleJDK 8 image (352MB 167MB) - frolvlad/alpine-oraclejkd8

[develar]

Thanks! I have successfully build alpine oracle JRE image — size 170MB (old, based on progrium/busybox — 162MB). https://registry.hub.docker.com/u/develar/java/

[CodingFabian]

hi @andyshinn this works like a charm, please let us know once you have merged it into a released alpine version!

[ifduyue]

Thanks @frol , we're using it now.

[frol]

@tpischke Then go and fork it. Just don't copy those broken manuals, please.

[tpischke]

I just copied the Dockerfiles from alpine-glibc and alpine-oraclejdk8 and built the images locally myself. Works beautifully. I'll discuss migration to OpenJDK with the team, but that's not going to happen in the near term. Testing alone will be a major effort if we migrate. For now I have a brilliant working solution, thanks to you guys. Alpine Docker Images are extremely compelling.

[jirutka]

Can the java version be specified? We test and recommend specific java versions to our customers.

No, only major version (there’s package openjdk7 and openjdk8). However, you can copy the openjdk8 abuild, adjust version (and patches if needed) and build it yourself. It’s quite easy and you’ll have it fully under your control.

'unofficial' external dependencies are forbidden by the project lead.

Then why are you trying to use that glibc package on Alpine? This is unofficial hack that is not supported by Alpine community. The only officially supported JRE/JDK are the mentioned openjdk packages built against musl libc.

Our project is very complex and getting it running on OpenJDK would likely be very costly...

Then you’re probably doing it wrong… Oracle JDK is just branded distribution of OpenJDK, the code base is nearly identical. That said, exceptions may exist, mainly in case of graphic applications.

If you really need OracleJDK, then use some glibc-based distribution, not Alpine. Adding glibc to Alpine is not a stable production-quality solution.

[frol]

@jirutka I also suggest to use OpenJDK wherever possible, but 60k pulls of my OracleJDK image suggest that it is stable enough, and I have been using it in production for quite a bit. Though, it depends on how you define the "production-quality" because Docker itself might be considered as not a production ready solution.

[tpischke]

We need to use Oracle Java, and it would sure be nice to use alpine linux. The fact that the official docker images are moving to alpine also plays a role in the decision. I hope alpine considers officially supporting oracle java in the future. For many java developers, openjdk is no alternative.

[jirutka]

…Docker itself might be considered as not a production ready solution.

Well, that’s very true! :)

I hope alpine considers officially supporting oracle java in the future.

Once again. Oracle JDK is a proprietary product, they don’t provide source-code of all their modifications to OpenJDK and build system, just binaries. These binaries are build against glibc (GNU C Library). Alpine Linux doesn’t use glibc, it’s based on (much better) musl libc.

Glibc and musl libc are (partially) incompatible (mainly because glibc doesn’t comply with POSIX standards). Therefore software built against glibc may not work on musl libc without recompiling. This is the case of Oracle JDK. Because it’s a proprietary product, we can’t recompile it and there’s nothing we can do about it!

libc is a very base system library, it’s not yet another library you can simply install along with others. Adding support for glibc would basically mean supporting another platform. The cost of this is too high. Also it doesn’t make much sense; Alpine with glibc would not be Alpine anymore! Alpine’s “motto” is: small, simply and secure. You can’t get this with glibc.

Ask Oracle for musl support, they have the key, not Alpine.

[ralph-tice]

Just as musl and glibc 'should' be interchangeable, the reality is that OpenJDK and Oracle JDK also 'should' be interchangeable. They're not, and it's irresponsible to ship hacks that enable glibc in a broken way on top of a musl-based system.

[jirutka]

Just as musl and glibc 'should' be interchangeable, the reality is that OpenJDK and Oracle JDK also 'should' be interchangeable.

This is very different situation. glibc and musl libc are different software, they both implements POSIX standards, but the code base is totally different. Oracle JDK is just a branded distribution of OpenJDK, the source code base is the same.
Also you can easily install more JDKs side by side on one system, but not two libc.

[ralph-tice]

You can't safely and stably install an Oracle JDK on a system with musl libc and it needs to be prominently admitted that such is the case instead of pushing these hacks forward.

You're also wrong about OracleJDK vs OpenJDK, there are significant differences and they do impact performance and stability. They can exist side by side with glibc provided to them, but the whole point of this exercise is that OracleJDK isn't compiled against musl libc.

[zaunerc]

I have contacted the Oracle support and I got the following statements regarding libmusl and the JRE / JDK:

The Oracle Java product organization maintains a limited number of "standard" configurations for Java and there is no plan to build it with the libmusl library. When customers request Java for other configurations, the Oracle Java product organization - Java Engineering Services (JES) - can create, certify and maintain custom version of Java for alternative O/S, special libraries, embedded architectures, older code bases, etc.

A typical engagement with JES would be: JES proposes a fixed-price engagement to do a "port" which is to deliver a certified version of Java on a non-standard platform. JES could also offer 12-months of Maintenance (renewable each year) to fix bugs and give updates.

[ncopa]

So basically, if you need OracleJDK, you'll have to use one of the "standard" configurations for Java (eg use windows or oracle linux). If you need something else you'll have to hire JES.

[leodutra]

"Most cross-compatible platform", they say...

The only exit would be a thin glibc facade bound to musl. And maybe this would be a kick for musl usage too.

Now we just need social engineering to convince a crazy guy for doing this.

[Denusdv]

We have a new glibc package if anyone would like to test this out. You can grab the package at https://circle-artifacts.com/gh/andyshinn/alpine-pkg-glibc/6/artifacts/0/home/ubuntu/alpine-pkg-glibc/packages/x86_64/glibc-2.21-r2.apk and install with apk add --allow-untrusted glibc-2.21-r2.apk. This should allow Oracle Java 8 and other glibc x86_64 native binaries to run on Alpine.

Do you provide the version compatible for ARM64 arch?

[jtlz2]

@Denusdv "no such project" at that link? :\

[sgerrand]

Hello, I took over maintenance of the glibc package from Andy. There are more recent versions available for download and installation at https://github.com/sgerrand/alpine-pkg-glibc/releases. I suggest you try there.

[realtimetodie]

https://bugs.openjdk.java.net/browse/JDK-8229469