jmap not happy on alpine #76

devinrsmith · 2016-05-25T20:30:11Z

We ran into an issue today where we needed to take a heap dump (jmap). But it wasn't working.

Image was java:openjdk-8u92-jdk-alpine.

$ docker exec server1_v3_1 ps aux
PID   USER     TIME   COMMAND
    1 root       1:54 java -server -verbose:gc ...
  121 root       0:00 ps aux

$ docker exec server1_v3_1 jmap -dump:live,format=b,file=/data/dump.map 1
1: Unable to get pid of LinuxThreads manager thread

Here's the source of the error: http://code.metager.de/source/xref/openjdk/jdk8/jdk/src/solaris/native/sun/tools/attach/LinuxVirtualMachine.c#255

We suspect it might be because alpine is using musl instead of glibc.

jmap works in java:openjdk-8u91-jdk.

The text was updated successfully, but these errors were encountered:

niloc132 · 2016-05-25T20:32:12Z

Other tools that don't work: jstack, jinfo, jhat.

paranoid945 · 2016-08-30T03:43:38Z

Not working in openjdk 7 neither.
does any openjdk engineer watching this issue?

yosifkit · 2016-09-01T19:47:23Z

This seems related: gliderlabs/docker-alpine#11.

michaelzg · 2017-01-15T06:43:19Z

Tried what @yosifkit linked.

Installed glibc as suggested by sgerrand in sgerrand/alpine-pkg-glibc#1 and still no luck:

/opt # apk info | grep glibc
glibc
glibc-bin

/opt # java -version
openjdk version "1.8.0_111-internal"
OpenJDK Runtime Environment (build 1.8.0_111-internal-alpine-r0-b14)
OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode)

/opt # ps aux
PID   USER     TIME   COMMAND
    1 root       8:38 java ...-jar ...
  303 root       0:00 /bin/sh
  421 root       0:00 ps aux

/opt # jmap -dump:format=b,file=heap.bin 1
1: Unable to get pid of LinuxThreads manager thread

Docker image: 8u111-jdk-alpine

tianon · 2017-01-27T00:43:25Z

Indeed -- I don't think there's much we can do about this. 😞

dkolog · 2017-01-31T08:39:54Z

I noticed that too. This happens because your java process PID is 1.
I use the same setup (alpine + glibc + jmap) and it works well if the PID is not 1 (when bash scripts forks the java process)

yosifkit · 2017-01-31T17:49:44Z

If you keep bash as PID 1, you'd no longer get signals from docker stop and docker kill (without having to write traps). With docker 1.13 you can use --init to have docker put in tini as PID 1; it'll forward signals and reap zombies.

dkolog · 2017-02-01T06:33:31Z

@yosifkit you are absolutely right.
@devinrsmith I was able to get heap/thread dump from PID 1 using Oracle's JDK.

elyscape · 2017-02-20T21:11:10Z

This may be a result of the debug symbols not being installed in the image. See comment 9 on this issue or comment 15 on this issue.

sgerrand · 2017-06-20T15:09:05Z

Installed glibc as suggested by @sgerrand

For those arriving late to the party, like me, this issue appears to be unrelated to the underlying libc implementation.

sitnik · 2017-07-05T12:10:18Z

I have to use docker 1.12 and don't know if '--init' will be used in Kubernetes, so I solved this issue by wrapping java process in shell script and adding trap for TERM signal. At least jmap now works for me.

tianon · 2018-01-03T21:30:47Z

Closing, since this appears to be solved by either using Docker's --init flag, or adding something like tini / dumb-init to images which suffer from this problem. 👍

ijaychang · 2019-01-23T02:09:16Z

So how can I use jmap ?

highel · 2019-02-27T09:58:07Z

So how can I use jmap ?

You can change docker file and instead of running java as java blahlahvlha run it as sh -c java blahlahvlha && 1 notice && 1 to prevent sh from exiting,

ScottOaks · 2019-06-25T20:26:12Z

I disagree that this is unrelated to the libc version in alpine.

A simple docker test with FROM openjdk:8 runs the java process as PID 1, and jcmd (and similar tools) still work. GIven this dockerfile
FROM openjdk:8
COPY . /tmp/hello
WORKDIR /tmp/hello
RUN javac Hello.java
CMD ["java", "Hello"]

building and running (assuming the Hello program is long-running), then I can do this:
$ docker exec mad_mahavira jcmd 1 info
...prints expected output....

Change the dockerfile to have FROM openjdk:8-alpine and only then do I have the issue of not being able to attach to PID 1.

I'm not saying this is definitely because of the use of musl instead of glibc; it could be some way that alpine linux sets up the /proc system that is different than other linux containers. But the issue does seem to lie with Alpine.

yashbhutwala · 2019-11-19T13:00:39Z

Ping! Still facing this issue, can we please reopen this and triage the underlying issue?

casimirrex · 2019-12-01T05:58:41Z

hi you need to add openjdk8.xx.alpine . we are getting the same problem.

yosifkit · 2019-12-03T00:59:52Z

Ping! Still facing this issue, can we please reopen this and triage the underlying issue?

No, OpenJDK 8 on Alpine are no longer maintained here since Alpine/musl in not officially supported by the OpenJDK project. See also #372 (comment).

After ~ 3 days of running, the success-modeling container runs out of memory (2Gb provided). [pool-10-thread-1] ERROR org.web3j.protocol.core.filters.Filter - Error sending request java.lang.OutOfMemoryError: Java heap space Possibly related, was nearby in the logs: WARNING: A connection to http://las2peer-ethnet:8545/ was leaked. Did you forget to close a response body? To see where this was allocated, set the OkHttpClient logger level to FINE: Logger.getLogger(OkHttpClient.class.getName()).setLevel(Level.FINE); see the following for details: - docker-library/openjdk#76 - http://www.programmersought.com/article/454192941/ - https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/ without running some sort of init process, we cannot create a heapdump of java to see where the memory leak occurs.

jhmartin · 2020-04-08T23:03:40Z

As a google breadcrumb; https://github.com/apangin/jattach is able to generate heap dumps from within an Alpine / OpenJDK8 based container.

This solves the issue related to docker-library/openjdk#76 In a container based on java alpine, commands like `jmap`, `jstack` don't work, and the pid of alluxio processes like `AlluxioWorker` is 1. Before: ``` bash-4.4$ jps 1 AlluxioMaster 904 Jps bash-4.4$ jstack 1 1: Unable to get pid of LinuxThreads manager thread ``` After: ``` bash-4.4# jps 357 Jps 6 AlluxioWorker # And jstack will work ``` pr-link: #12233 change-id: cid-e60f913ec30d1718142d1311eb97e4f44b568973

ccgcool · 2021-09-25T11:22:36Z

I too faced this issue with openjdk:8-jdk-alpine when PID was 1. Then I changed base image to maven:3.8.2-jdk-8 . Then It worked fine even with PID 1. So its likely not a PID being 1 issue. note: The image which solved the issue is a jdk8 based image by maven, not an openjdk based image. For openjdk you may have to install the openjdk-devel or openjdk-dbg kinda stuff because openjdk apparently does not come with debug facilities like jmap stack etc

cliveverghese · 2021-12-02T20:07:48Z

This issue is fixed by patch icedtea-issue13032.patch and on amazoncorretto:8-alpine as well.

robberphex · 2022-12-03T16:18:37Z

TL; DR:

pid=1 ;\
touch /proc/${pid}/cwd/.attach_pid${pid} && \
  kill -SIGQUIT ${pid} && \
  sleep 2 &&
  ls /proc/${pid}/root/tmp/.java_pid${pid}
# Attach Listener started, then you can attach this jvm normally.

The detail analysis is on the post(Chinese).

bookc-man · 2023-03-22T09:45:47Z

run cmd add --init is ok

docker run --init

cyao2q · 2023-06-13T02:47:42Z

run cmd add --init is ok

docker run --init

Dockerfile中如何编写？

bookc-man · 2023-06-13T02:52:57Z

run cmd add --init is ok
docker run --init

Dockerfile中如何编写？

正常写就行了
例如
`
FROM openjdk-8-alpine:latest

COPY ../target/management.jar /opt/project/management.jar

WORKDIR /opt/project

CMD java -jar management.jar
`

cyao2q · 2023-06-13T03:17:29Z

Dockerfile中如何使用 --init 参数还是只能docker run的时候使用

目前可以使用 tini 插件解决就是感觉不想安装

FROM openjdk:8-jdk-alpine3.9

COPY *.jar /app.jar

RUN apk --no-cache add tini

ENTRYPOINT tini java -Dfile.encoding=UTF-8 -Duser.timezone=PRC -Dserver.port=8080 -jar /app.jar

bookc-man · 2023-06-13T03:18:55Z

Dockerfile中如何使用 --init 参数还是只能docker run的时候使用

目前可以使用 tini 插件解决就是感觉不想安装
FROM openjdk:8-jdk-alpine3.9

COPY *.jar /app.jar

RUN apk --no-cache add tini

CMD java -Dfile.encoding=UTF-8 -Duser.timezone=PRC -Dserver.port=8080 -jar /app.jar

ENTRYPOINT ["tini"]

直接在docker run时候添加参数就行，不需要安装其他插件

cyao2q · 2023-06-13T03:19:27Z

Dockerfile中如何使用 --init 参数还是只能docker run的时候使用
目前可以使用 tini 插件解决就是感觉不想安装
FROM openjdk:8-jdk-alpine3.9

COPY *.jar /app.jar

RUN apk --no-cache add tini

CMD java -Dfile.encoding=UTF-8 -Duser.timezone=PRC -Dserver.port=8080 -jar /app.jar

ENTRYPOINT ["tini"]
直接在docker run时候添加参数就行，不需要安装其他插件

现在是在K8S运行那样可能需要运维修改了

tianon closed this as completed Jan 3, 2018

sonix07 mentioned this issue Feb 12, 2018

constant amount of underreplicated records crate/crate#6887

Closed

ghost mentioned this issue Apr 6, 2018

Do we need a init system? navikt/baseimages#2

Closed

rbraley mentioned this issue Jun 28, 2018

Jstack and other debugging tools not accessible using rp-start lightbend/sbt-reactive-app#118

Closed

trask mentioned this issue Oct 16, 2018

Heap histogram inside Alpine-based Docker container glowroot/glowroot#465

Closed

hengyunabc mentioned this issue Dec 18, 2018

Attach docker 里 pid 1的进程报错 alibaba/arthas#362

Closed

1 task

yashbhutwala mentioned this issue Nov 19, 2019

Not able to run jmap, jstack, jcmd on alpine #372

Closed

slupczynskim mentioned this issue Dec 16, 2019

added tini for init management rwth-acis/mobsos-success-modeling#7

Merged

goudai mentioned this issue Apr 6, 2020

How to starting project using Alpine's Tini ？ GoogleContainerTools/jib#2386

Closed

jiacheliu3 mentioned this issue Oct 12, 2020

Run alluxio processes with tini in containers Alluxio/alluxio#12233

Merged

alluxio-bot mentioned this issue Oct 15, 2020

Port changes of [#12233] to branch-2.4 Alluxio/alluxio#12309

Merged

hengyunabc mentioned this issue Mar 12, 2021

elasticsearch docker image problem elastic/elasticsearch#66487

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jmap not happy on alpine #76

jmap not happy on alpine #76

devinrsmith commented May 25, 2016 •

edited

Loading

niloc132 commented May 25, 2016

paranoid945 commented Aug 30, 2016 •

edited

Loading

yosifkit commented Sep 1, 2016

michaelzg commented Jan 15, 2017 •

edited

Loading

tianon commented Jan 27, 2017

dkolog commented Jan 31, 2017

yosifkit commented Jan 31, 2017

dkolog commented Feb 1, 2017

elyscape commented Feb 20, 2017

sgerrand commented Jun 20, 2017

sitnik commented Jul 5, 2017

tianon commented Jan 3, 2018

ijaychang commented Jan 23, 2019

highel commented Feb 27, 2019 •

edited

Loading

ScottOaks commented Jun 25, 2019

yashbhutwala commented Nov 19, 2019 •

edited

Loading

casimirrex commented Dec 1, 2019

yosifkit commented Dec 3, 2019

jhmartin commented Apr 8, 2020

ccgcool commented Sep 25, 2021 •

edited

Loading

cliveverghese commented Dec 2, 2021

robberphex commented Dec 3, 2022

bookc-man commented Mar 22, 2023

cyao2q commented Jun 13, 2023

bookc-man commented Jun 13, 2023

cyao2q commented Jun 13, 2023 •

edited

Loading

bookc-man commented Jun 13, 2023

cyao2q commented Jun 13, 2023

jmap not happy on alpine #76

jmap not happy on alpine #76

Comments

devinrsmith commented May 25, 2016 • edited Loading

niloc132 commented May 25, 2016

paranoid945 commented Aug 30, 2016 • edited Loading

yosifkit commented Sep 1, 2016

michaelzg commented Jan 15, 2017 • edited Loading

tianon commented Jan 27, 2017

dkolog commented Jan 31, 2017

yosifkit commented Jan 31, 2017

dkolog commented Feb 1, 2017

elyscape commented Feb 20, 2017

sgerrand commented Jun 20, 2017

sitnik commented Jul 5, 2017

tianon commented Jan 3, 2018

ijaychang commented Jan 23, 2019

highel commented Feb 27, 2019 • edited Loading

ScottOaks commented Jun 25, 2019

yashbhutwala commented Nov 19, 2019 • edited Loading

casimirrex commented Dec 1, 2019

yosifkit commented Dec 3, 2019

jhmartin commented Apr 8, 2020

ccgcool commented Sep 25, 2021 • edited Loading

cliveverghese commented Dec 2, 2021

robberphex commented Dec 3, 2022

bookc-man commented Mar 22, 2023

cyao2q commented Jun 13, 2023

bookc-man commented Jun 13, 2023

cyao2q commented Jun 13, 2023 • edited Loading

bookc-man commented Jun 13, 2023

cyao2q commented Jun 13, 2023

devinrsmith commented May 25, 2016 •

edited

Loading

paranoid945 commented Aug 30, 2016 •

edited

Loading

michaelzg commented Jan 15, 2017 •

edited

Loading

highel commented Feb 27, 2019 •

edited

Loading

yashbhutwala commented Nov 19, 2019 •

edited

Loading

ccgcool commented Sep 25, 2021 •

edited

Loading

cyao2q commented Jun 13, 2023 •

edited

Loading