Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java 11 native image builds are about 20% slower #2129

Closed
dmlloyd opened this issue Feb 4, 2020 · 23 comments
Closed

Java 11 native image builds are about 20% slower #2129

dmlloyd opened this issue Feb 4, 2020 · 23 comments
Assignees

Comments

@dmlloyd
Copy link
Contributor

dmlloyd commented Feb 4, 2020

Describe GraalVM and your environment :

  • GraalVM version or commit id if built from source: 19.3.1
  • CE or EE: CE
  • Build Time or run time failure: Build time
  • JDK version: JDK 8 vs JDK 11
  • Native compiler information:
gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  • Native linker information:
collect2 version 8.3.1 20190223 (Red Hat 8.3.1-2)
/usr/bin/ld -plugin /usr/libexec/gcc/x86_64-redhat-linux/8/liblto_plugin.so -plugin-opt=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper -plugin-opt=-fresolution=/tmp/ccaVDHOC.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crt1.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crti.o /usr/lib/gcc/x86_64-redhat-linux/8/crtbegin.o -L/usr/lib/gcc/x86_64-redhat-linux/8 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/8/../../.. --version -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/x86_64-redhat-linux/8/crtend.o /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64/crtn.o
GNU ld version 2.31.1-25.fc29
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
  • OS and OS Version: Linux 5.1.11-200.fc29.x86_64 #1 SMP Mon Jun 17 19:30:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Architecture: AMD64
  • The output of java -Xinternalversion:
OpenJDK 64-Bit Server VM (11.0.6+9-jvmci-19.3-b07) for linux-amd64 JRE (11.0.6+9-jvmci-19.3-b07), built on Jan 13 2020 20:43:28 by "buildslave" with gcc 7.3.0

and

OpenJDK 64-Bit GraalVM CE 19.3.1 (25.242-b06-jvmci-19.3-b07) for linux-amd64 JRE (8u242), built on Jan 13 2020 23:00:49 by "buildslave" with gcc 7.3.0

Have you verified this issue still happens when using the latest snapshot?
Not yet.

Describe the issue
A repeated build of the same project takes about 20% longer when building for Java 11 than Java 8. Practically speaking, this is causing capacity issues on our CI.

Describe the full native-image command

native-image -J-Dsun.nio.ch.maxUpdateArraySize=100 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Dio.netty.leakDetection.level=DISABLED -J-Dio.netty.allocator.maxOrder=1 -J-Dvertx.logger-delegate-factory-class-name=io.quarkus.vertx.core.runtime.VertxLogDelegateFactory -J-Dvertx.disableDnsResolver=true --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -H:+JNI -jar validation-quickstart-1.0-SNAPSHOT-runner.jar -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:NativeLinkerOption=-no-pie --no-server -H:-UseServiceLoaderFeature -H:+StackTrace validation-quickstart-1.0-SNAPSHOT-runner

Code snippet or code repository that reproduces the issue

In order to characterize this issue in a rational way, we've reproduced the issue in the context of an existing example Quarkus-based project built into a native image. Here are the steps to reproduce my results. Requires Maven 3.6.0 and Git.

  • git clone [email protected]:quarkusio/quarkus.git
  • cd quarkus
  • git checkout a63bba571d2a7b0d9f569b8a2a9df369f8378137
  • mvn install -DskipTests -DskipITs

In a new directory:

  • git clone [email protected]:quarkusio/quarkus-quickstarts.git
  • cd quarkus-quickstarts/validation-quickstart
  • Ensure that the GraalVM's java and native-image are in your PATH
  • Ensure that the GRAALVM_HOME and JAVA_HOME env vars point to the GraalVM instance being tested (8 or 11)
  • mvn clean -Dquarkus.version=999-SNAPSHOT
  • mvn install -Dquarkus.version=999-SNAPSHOT -Dnative

Expected behavior
It is expected that build times between Java 8 and 11 be largely similar.

Additional context

Viewing the build timings that are displayed by native-image, it seems that there isn't one single place where things run slower. Rather it's fairly evenly distributed. So, there might be some overall problem like GC having an effect.

We're going to try to gather additional data around things like memory usage and so forth.

Here's a graph showing total time across 5 runs:

Build time comparison graph

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 4, 2020

@christianwimmer
Copy link

JDK 11 changed the default GC from parallel GC to G1. Please check if that is the reason for the difference.

@pejovica We talked about switching the default GC to the parallel GC on JDK 11, but I think that is not done yet?

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 4, 2020

I did one run with parallel GC on 11 and it was only slightly better than the original run (76,412.91 ms). So it's not that - or rather, it's not just that. It's possible that there are some defaults that need to be adjusted in terms of memory sizes and that sort of thing. I'm trying to figure out what the defaults are on the 8 build so I can replicated them for 11.

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 4, 2020

Here is a sample comparison of 8 versus 11 in terms of GC events. 8 is running with -XX:+UseParallelGC, and both are running with -XX:+PrintGCDetails.

Java 8

[GC (Allocation Failure) [PSYoungGen: 262144K->18867K(305664K)] 262144K->18891K(1005056K), 0.0115006 secs] [Times: user=0.04 sys=0.01, real=0.01 secs] 
[GC (Metadata GC Threshold) [PSYoungGen: 84987K->18012K(305664K)] 85011K->18044K(1005056K), 0.0083377 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
[Full GC (Metadata GC Threshold) [PSYoungGen: 18012K->0K(305664K)] [ParOldGen: 32K->17665K(699392K)] 18044K->17665K(1005056K), [Metaspace: 20782K->20770K(1069056K)], 0.0376293 secs] [Times: user=0.16 sys=0.01, real=0.04 secs] 
[GC (Allocation Failure) [PSYoungGen: 262144K->16870K(305664K)] 279809K->34544K(1005056K), 0.0088389 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
[GC (Metadata GC Threshold) [PSYoungGen: 56051K->12171K(419840K)] 73724K->29845K(1119232K), 0.0075270 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
[Full GC (Metadata GC Threshold) [PSYoungGen: 12171K->0K(419840K)] [ParOldGen: 17673K->27042K(699392K)] 29845K->27042K(1119232K), [Metaspace: 34782K->34782K(1079296K)], 0.0735204 secs] [Times: user=0.37 sys=0.02, real=0.07 secs] 
[GC (Allocation Failure) [PSYoungGen: 376320K->23794K(419840K)] 403362K->50844K(1119232K), 0.0149071 secs] [Times: user=0.06 sys=0.01, real=0.01 secs] 
[GC (Metadata GC Threshold) [PSYoungGen: 230889K->25065K(539136K)] 257939K->52872K(1238528K), 0.0146178 secs] [Times: user=0.05 sys=0.01, real=0.02 secs] 
[Full GC (Metadata GC Threshold) [PSYoungGen: 25065K->0K(539136K)] [ParOldGen: 27807K->48218K(928256K)] 52872K->48218K(1467392K), [Metaspace: 58230K->58223K(1099776K)], 0.1462430 secs] [Times: user=0.65 sys=0.02, real=0.14 secs] 
[validation-quickstart-1.0-SNAPSHOT-runner:19886]    classlist:   5,369.80 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]        (cap):     809.24 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]        setup:   2,254.28 ms
[GC (Allocation Failure) [PSYoungGen: 514048K->28644K(542720K)] 562266K->88938K(1470976K), 0.0334478 secs] [Times: user=0.15 sys=0.02, real=0.04 secs] 
11:08:29,950 INFO  [org.hib.val.int.uti.Version] HV000001: Hibernate Validator 6.1.2.Final
11:08:31,287 INFO  [org.jbo.threads] JBoss Threads version 3.0.1.Final
[GC (Allocation Failure) [PSYoungGen: 542692K->37856K(658432K)] 602986K->178502K(1586688K), 0.0846542 secs] [Times: user=0.50 sys=0.03, real=0.09 secs] 
[GC (Allocation Failure) [PSYoungGen: 658400K->78318K(698880K)] 799046K->296294K(1627136K), 0.1198603 secs] [Times: user=0.72 sys=0.06, real=0.12 secs] 
[GC (Metadata GC Threshold) [PSYoungGen: 461530K->122350K(838656K)] 679507K->366226K(1766912K), 0.1338656 secs] [Times: user=0.70 sys=0.05, real=0.14 secs] 
[Full GC (Metadata GC Threshold) [PSYoungGen: 122350K->0K(838656K)] [ParOldGen: 243876K->346510K(1610240K)] 366226K->346510K(2448896K), [Metaspace: 95735K->95646K(1132544K)], 1.1169239 secs] [Times: user=5.82 sys=0.05, real=1.12 secs] 
[GC (Allocation Failure) [PSYoungGen: 716288K->125293K(868864K)] 1062798K->471811K(2479104K), 0.0731976 secs] [Times: user=0.51 sys=0.03, real=0.07 secs] 
[GC (Allocation Failure) [PSYoungGen: 841581K->167935K(983552K)] 1188099K->583413K(2593792K), 0.1422186 secs] [Times: user=0.87 sys=0.08, real=0.14 secs] 
[GC (Allocation Failure) [PSYoungGen: 983551K->221669K(1037312K)] 1399029K->676245K(2647552K), 0.1747571 secs] [Times: user=1.10 sys=0.05, real=0.17 secs] 
[GC (Allocation Failure) [PSYoungGen: 1037285K->132435K(1299456K)] 1491861K->736829K(2909696K), 0.1632846 secs] [Times: user=1.14 sys=0.10, real=0.16 secs] 
[GC (Allocation Failure) [PSYoungGen: 1158483K->152486K(1294848K)] 1762877K->756880K(2905088K), 0.0935858 secs] [Times: user=0.61 sys=0.00, real=0.09 secs] 
[validation-quickstart-1.0-SNAPSHOT-runner:19886]   (typeflow):  14,412.16 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]    (objects):   9,004.03 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]   (features):     481.85 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]     analysis:  24,948.82 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]     (clinit):     707.82 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]     universe:   1,791.68 ms
[GC (Allocation Failure) [PSYoungGen: 1178534K->210877K(1533952K)] 1782928K->815279K(3144192K), 0.1232042 secs] [Times: user=0.85 sys=0.08, real=0.13 secs] 
[validation-quickstart-1.0-SNAPSHOT-runner:19886]      (parse):   2,285.26 ms
[GC (Allocation Failure) [PSYoungGen: 1473981K->292840K(1555968K)] 2078383K->1172448K(3166208K), 0.3046454 secs] [Times: user=1.89 sys=0.16, real=0.30 secs] 
[GC (Allocation Failure) [PSYoungGen: 1555944K->360278K(1883648K)] 2435552K->1252339K(3493888K), 0.2226393 secs] [Times: user=1.37 sys=0.13, real=0.23 secs] 
[validation-quickstart-1.0-SNAPSHOT-runner:19886]     (inline):   4,192.06 ms
[GC (Allocation Failure) [PSYoungGen: 1802070K->333887K(1928704K)] 2694131K->1260700K(3538944K), 0.1795830 secs] [Times: user=1.19 sys=0.02, real=0.18 secs] 
[GC (Allocation Failure) [PSYoungGen: 1775679K->27774K(2248704K)] 2702492K->1278567K(3858944K), 0.1868443 secs] [Times: user=1.01 sys=0.13, real=0.19 secs] 
[GC (Allocation Failure) [PSYoungGen: 1772158K->22166K(2254336K)] 3022951K->1298967K(3864576K), 0.0284516 secs] [Times: user=0.16 sys=0.01, real=0.03 secs] 
[GC (Allocation Failure) [PSYoungGen: 1766550K->42121K(2466816K)] 3043351K->1318931K(4077056K), 0.0256532 secs] [Times: user=0.16 sys=0.02, real=0.03 secs] 
[GC (Allocation Failure) [PSYoungGen: 2070665K->66316K(2519552K)] 3347475K->1343134K(4129792K), 0.0338586 secs] [Times: user=0.21 sys=0.00, real=0.04 secs] 
[GC (Allocation Failure) [PSYoungGen: 2094860K->91298K(2696704K)] 3371678K->1368123K(4306944K), 0.0455713 secs] [Times: user=0.26 sys=0.03, real=0.04 secs] 
[GC (Allocation Failure) [PSYoungGen: 2388642K->119472K(2745344K)] 3665467K->1396306K(4355584K), 0.0565631 secs] [Times: user=0.31 sys=0.06, real=0.06 secs] 
[validation-quickstart-1.0-SNAPSHOT-runner:19886]    (compile):  19,337.43 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]      compile:  27,369.60 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]        image:   2,087.85 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]        write:     420.62 ms
[validation-quickstart-1.0-SNAPSHOT-runner:19886]      [total]:  64,696.99 ms
Heap
 PSYoungGen      total 2745344K, used 1327332K [0x0000000695580000, 0x0000000762180000, 0x00000007c0000000)
  eden space 2297344K, 52% used [0x0000000695580000,0x00000006df10cdc8,0x0000000721900000)
  from space 448000K, 26% used [0x0000000721900000,0x0000000728dac240,0x000000073ce80000)
  to   space 420352K, 0% used [0x0000000748700000,0x0000000748700000,0x0000000762180000)
 ParOldGen       total 1610240K, used 1276833K [0x0000000440000000, 0x00000004a2480000, 0x0000000695580000)
  object space 1610240K, 79% used [0x0000000440000000,0x000000048dee8720,0x00000004a2480000)
 Metaspace       used 107813K, capacity 113351K, committed 113664K, reserved 1146880K
  class space    used 15908K, capacity 16772K, committed 16896K, reserved 1048576K

Java 11:

[0.001s][warning][gc] -XX:+PrintGCDetails is deprecated. Will use -Xlog:gc* instead.
[0.005s][info   ][gc] Using Parallel
[0.005s][info   ][gc,heap,coops] Heap address: 0x0000000480000000, size: 14336 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
[0.928s][info   ][gc,start     ] GC(0) Pause Young (Metadata GC Threshold)
[0.940s][info   ][gc,heap      ] GC(0) PSYoungGen: 157784K->21293K(305664K)
[0.940s][info   ][gc,heap      ] GC(0) ParOldGen: 0K->24K(699392K)
[0.940s][info   ][gc,metaspace ] GC(0) Metaspace: 20716K->20716K(1067008K)
[0.940s][info   ][gc           ] GC(0) Pause Young (Metadata GC Threshold) 154M->20M(981M) 12.063ms
[0.940s][info   ][gc,cpu       ] GC(0) User=0.06s Sys=0.01s Real=0.02s
[0.940s][info   ][gc,start     ] GC(1) Pause Full (Metadata GC Threshold)
[0.940s][info   ][gc,phases,start] GC(1) Marking Phase
[0.953s][info   ][gc,phases      ] GC(1) Marking Phase 12.667ms
[0.953s][info   ][gc,phases,start] GC(1) Summary Phase
[0.953s][info   ][gc,phases      ] GC(1) Summary Phase 0.007ms
[0.953s][info   ][gc,phases,start] GC(1) Adjust Roots
[0.960s][info   ][gc,phases      ] GC(1) Adjust Roots 7.716ms
[0.960s][info   ][gc,phases,start] GC(1) Compaction Phase
[0.983s][info   ][gc,phases      ] GC(1) Compaction Phase 22.309ms
[0.983s][info   ][gc,phases,start] GC(1) Post Compact
[0.983s][info   ][gc,phases      ] GC(1) Post Compact 0.385ms
[0.983s][info   ][gc,heap        ] GC(1) PSYoungGen: 21293K->0K(305664K)
[0.983s][info   ][gc,heap        ] GC(1) ParOldGen: 24K->20815K(699392K)
[0.983s][info   ][gc,metaspace   ] GC(1) Metaspace: 20716K->20654K(1067008K)
[0.983s][info   ][gc             ] GC(1) Pause Full (Metadata GC Threshold) 20M->20M(981M) 43.252ms
[0.983s][info   ][gc,cpu         ] GC(1) User=0.20s Sys=0.01s Real=0.04s
[1.602s][info   ][gc,start       ] GC(2) Pause Young (Metadata GC Threshold)
[1.609s][info   ][gc,heap        ] GC(2) PSYoungGen: 79851K->11683K(305664K)
[1.609s][info   ][gc,heap        ] GC(2) ParOldGen: 20815K->20815K(699392K)
[1.609s][info   ][gc,metaspace   ] GC(2) Metaspace: 34822K->34822K(1079296K)
[1.609s][info   ][gc             ] GC(2) Pause Young (Metadata GC Threshold) 98M->31M(981M) 7.626ms
[1.609s][info   ][gc,cpu         ] GC(2) User=0.04s Sys=0.01s Real=0.01s
[1.609s][info   ][gc,start       ] GC(3) Pause Full (Metadata GC Threshold)
[1.609s][info   ][gc,phases,start] GC(3) Marking Phase
[1.629s][info   ][gc,phases      ] GC(3) Marking Phase 19.057ms
[1.629s][info   ][gc,phases,start] GC(3) Summary Phase
[1.629s][info   ][gc,phases      ] GC(3) Summary Phase 0.009ms
[1.629s][info   ][gc,phases,start] GC(3) Adjust Roots
[1.641s][info   ][gc,phases      ] GC(3) Adjust Roots 12.550ms
[1.641s][info   ][gc,phases,start] GC(3) Compaction Phase
[1.720s][info   ][gc,phases      ] GC(3) Compaction Phase 78.373ms
[1.720s][info   ][gc,phases,start] GC(3) Post Compact
[1.720s][info   ][gc,phases      ] GC(3) Post Compact 0.146ms
[1.720s][info   ][gc,heap        ] GC(3) PSYoungGen: 11683K->0K(305664K)
[1.720s][info   ][gc,heap        ] GC(3) ParOldGen: 20815K->31145K(699392K)
[1.720s][info   ][gc,metaspace   ] GC(3) Metaspace: 34822K->34819K(1079296K)
[1.720s][info   ][gc             ] GC(3) Pause Full (Metadata GC Threshold) 31M->30M(981M) 110.290ms
[1.720s][info   ][gc,cpu         ] GC(3) User=0.58s Sys=0.00s Real=0.11s
[3.109s][info   ][gc,start       ] GC(4) Pause Young (Metadata GC Threshold)
[3.119s][info   ][gc,heap        ] GC(4) PSYoungGen: 261970K->18348K(305664K)
[3.119s][info   ][gc,heap        ] GC(4) ParOldGen: 31145K->31145K(699392K)
[3.119s][info   ][gc,metaspace   ] GC(4) Metaspace: 58365K->58365K(1099776K)
[3.119s][info   ][gc             ] GC(4) Pause Young (Metadata GC Threshold) 286M->48M(981M) 10.089ms
[3.119s][info   ][gc,cpu         ] GC(4) User=0.06s Sys=0.00s Real=0.01s
[3.119s][info   ][gc,start       ] GC(5) Pause Full (Metadata GC Threshold)
[3.119s][info   ][gc,phases,start] GC(5) Marking Phase
[3.157s][info   ][gc,phases      ] GC(5) Marking Phase 37.798ms
[3.157s][info   ][gc,phases,start] GC(5) Summary Phase
[3.157s][info   ][gc,phases      ] GC(5) Summary Phase 0.007ms
[3.157s][info   ][gc,phases,start] GC(5) Adjust Roots
[3.175s][info   ][gc,phases      ] GC(5) Adjust Roots 18.003ms
[3.175s][info   ][gc,phases,start] GC(5) Compaction Phase
[3.288s][info   ][gc,phases      ] GC(5) Compaction Phase 112.781ms
[3.288s][info   ][gc,phases,start] GC(5) Post Compact
[3.288s][info   ][gc,phases      ] GC(5) Post Compact 0.200ms
[3.288s][info   ][gc,heap        ] GC(5) PSYoungGen: 18348K->0K(305664K)
[3.288s][info   ][gc,heap        ] GC(5) ParOldGen: 31145K->47112K(699392K)
[3.288s][info   ][gc,metaspace   ] GC(5) Metaspace: 58365K->58350K(1099776K)
[3.288s][info   ][gc             ] GC(5) Pause Full (Metadata GC Threshold) 48M->46M(981M) 168.928ms
[3.288s][info   ][gc,cpu         ] GC(5) User=0.86s Sys=0.01s Real=0.16s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]    classlist:   4,189.09 ms
[5.663s][info   ][gc,start       ] GC(6) Pause Young (Allocation Failure)
[5.681s][info   ][gc,heap        ] GC(6) PSYoungGen: 262144K->32933K(305664K)
[5.681s][info   ][gc,heap        ] GC(6) ParOldGen: 47112K->47120K(699392K)
[5.681s][info   ][gc,metaspace   ] GC(6) Metaspace: 91095K->91095K(1126400K)
[5.682s][info   ][gc             ] GC(6) Pause Young (Allocation Failure) 302M->78M(981M) 18.910ms
[5.682s][info   ][gc,cpu         ] GC(6) User=0.11s Sys=0.00s Real=0.02s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]        (cap):     772.36 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]        setup:   2,197.57 ms
11:04:36,138 INFO  [org.hib.val.int.uti.Version] HV000001: Hibernate Validator 6.1.2.Final
[7.637s][info   ][gc,start       ] GC(7) Pause Young (Metadata GC Threshold)
[7.662s][info   ][gc,heap        ] GC(7) PSYoungGen: 156080K->43507K(305664K)
[7.662s][info   ][gc,heap        ] GC(7) ParOldGen: 47120K->51208K(699392K)
[7.662s][info   ][gc,metaspace   ] GC(7) Metaspace: 96745K->96745K(1130496K)
[7.662s][info   ][gc             ] GC(7) Pause Young (Metadata GC Threshold) 198M->92M(981M) 24.894ms
[7.662s][info   ][gc,cpu         ] GC(7) User=0.15s Sys=0.01s Real=0.03s
[7.662s][info   ][gc,start       ] GC(8) Pause Full (Metadata GC Threshold)
[7.662s][info   ][gc,phases,start] GC(8) Marking Phase
[7.736s][info   ][gc,phases      ] GC(8) Marking Phase 73.245ms
[7.736s][info   ][gc,phases,start] GC(8) Summary Phase
[7.736s][info   ][gc,phases      ] GC(8) Summary Phase 0.026ms
[7.736s][info   ][gc,phases,start] GC(8) Adjust Roots
[7.775s][info   ][gc,phases      ] GC(8) Adjust Roots 39.353ms
[7.775s][info   ][gc,phases,start] GC(8) Compaction Phase
[7.888s][info   ][gc,phases      ] GC(8) Compaction Phase 112.808ms
[7.888s][info   ][gc,phases,start] GC(8) Post Compact
[7.889s][info   ][gc,phases      ] GC(8) Post Compact 1.207ms
[7.889s][info   ][gc,heap        ] GC(8) PSYoungGen: 43507K->0K(305664K)
[7.889s][info   ][gc,heap        ] GC(8) ParOldGen: 51208K->90117K(699392K)
[7.889s][info   ][gc,metaspace   ] GC(8) Metaspace: 96745K->96634K(1130496K)
[7.889s][info   ][gc             ] GC(8) Pause Full (Metadata GC Threshold) 92M->88M(981M) 226.830ms
[7.889s][info   ][gc,cpu         ] GC(8) User=0.97s Sys=0.02s Real=0.23s
11:04:37,527 INFO  [org.jbo.threads] JBoss Threads version 3.0.1.Final
[9.584s][info   ][gc,start       ] GC(9) Pause Young (Allocation Failure)
[9.617s][info   ][gc,heap        ] GC(9) PSYoungGen: 262144K->46752K(304640K)
[9.617s][info   ][gc,heap        ] GC(9) ParOldGen: 90117K->90125K(699392K)
[9.617s][info   ][gc,metaspace   ] GC(9) Metaspace: 98620K->98620K(1132544K)
[9.617s][info   ][gc             ] GC(9) Pause Young (Allocation Failure) 344M->133M(980M) 32.398ms
[9.617s][info   ][gc,cpu         ] GC(9) User=0.20s Sys=0.00s Real=0.03s
[10.877s][info   ][gc,start       ] GC(10) Pause Young (Allocation Failure)
[10.946s][info   ][gc,heap        ] GC(10) PSYoungGen: 301216K->57824K(312320K)
[10.946s][info   ][gc,heap        ] GC(10) ParOldGen: 90125K->120806K(699392K)
[10.946s][info   ][gc,metaspace   ] GC(10) Metaspace: 99505K->99505K(1134592K)
[10.946s][info   ][gc             ] GC(10) Pause Young (Allocation Failure) 382M->174M(988M) 69.232ms
[10.946s][info   ][gc,cpu         ] GC(10) User=0.36s Sys=0.01s Real=0.07s
[12.004s][info   ][gc,start       ] GC(11) Pause Young (Allocation Failure)
[12.075s][info   ][gc,heap        ] GC(11) PSYoungGen: 312288K->83445K(569344K)
[12.075s][info   ][gc,heap        ] GC(11) ParOldGen: 120806K->140032K(699392K)
[12.075s][info   ][gc,metaspace   ] GC(11) Metaspace: 100310K->100310K(1134592K)
[12.075s][info   ][gc             ] GC(11) Pause Young (Allocation Failure) 422M->218M(1239M) 71.085ms
[12.075s][info   ][gc,cpu         ] GC(11) User=0.43s Sys=0.04s Real=0.07s
[14.109s][info   ][gc,start       ] GC(12) Pause Young (Allocation Failure)
[14.249s][info   ][gc,heap        ] GC(12) PSYoungGen: 569333K->106482K(592384K)
[14.250s][info   ][gc,heap        ] GC(12) ParOldGen: 140032K->196935K(699392K)
[14.250s][info   ][gc,metaspace   ] GC(12) Metaspace: 101811K->101811K(1136640K)
[14.250s][info   ][gc             ] GC(12) Pause Young (Allocation Failure) 692M->296M(1261M) 140.606ms
[14.250s][info   ][gc,cpu         ] GC(12) User=0.73s Sys=0.06s Real=0.14s
[15.761s][info   ][gc,start       ] GC(13) Pause Young (Allocation Failure)
[15.896s][info   ][gc,heap        ] GC(13) PSYoungGen: 592370K->129940K(906752K)
[15.896s][info   ][gc,heap        ] GC(13) ParOldGen: 196935K->246225K(699392K)
[15.896s][info   ][gc,metaspace   ] GC(13) Metaspace: 102796K->102796K(1136640K)
[15.896s][info   ][gc             ] GC(13) Pause Young (Allocation Failure) 770M->367M(1568M) 135.174ms
[15.896s][info   ][gc,cpu         ] GC(13) User=0.93s Sys=0.07s Real=0.13s
[18.714s][info   ][gc,start       ] GC(14) Pause Young (Allocation Failure)
[18.903s][info   ][gc,heap        ] GC(14) PSYoungGen: 889748K->101794K(924672K)
[18.903s][info   ][gc,heap        ] GC(14) ParOldGen: 246225K->371559K(699392K)
[18.903s][info   ][gc,metaspace   ] GC(14) Metaspace: 107193K->107193K(1140736K)
[18.903s][info   ][gc             ] GC(14) Pause Young (Allocation Failure) 1109M->462M(1586M) 189.040ms
[18.903s][info   ][gc,cpu         ] GC(14) User=1.02s Sys=0.08s Real=0.19s
[21.117s][info   ][gc,start       ] GC(15) Pause Young (Allocation Failure)
[21.245s][info   ][gc,heap        ] GC(15) PSYoungGen: 861602K->106062K(1380864K)
[21.245s][info   ][gc,heap        ] GC(15) ParOldGen: 371559K->462457K(699392K)
[21.245s][info   ][gc,metaspace   ] GC(15) Metaspace: 108124K->108124K(1140736K)
[21.245s][info   ][gc             ] GC(15) Pause Young (Allocation Failure) 1204M->555M(2031M) 128.037ms
[21.245s][info   ][gc,cpu         ] GC(15) User=0.77s Sys=0.07s Real=0.13s
[24.078s][info   ][gc,start       ] GC(16) Pause Young (Allocation Failure)
[24.245s][info   ][gc,heap        ] GC(16) PSYoungGen: 1318478K->147339K(1384448K)
[24.245s][info   ][gc,heap        ] GC(16) ParOldGen: 462457K->554074K(699392K)
[24.245s][info   ][gc,metaspace   ] GC(16) Metaspace: 109344K->109344K(1142784K)
[24.245s][info   ][gc             ] GC(16) Pause Young (Allocation Failure) 1739M->684M(2035M) 167.339ms
[24.245s][info   ][gc,cpu         ] GC(16) User=0.92s Sys=0.09s Real=0.17s
[27.319s][info   ][gc,start       ] GC(17) Pause Young (Allocation Failure)
[27.439s][info   ][gc,heap        ] GC(17) PSYoungGen: 1359755K->73172K(2109952K)
[27.439s][info   ][gc,heap        ] GC(17) ParOldGen: 554074K->678829K(699392K)
[27.439s][info   ][gc,metaspace   ] GC(17) Metaspace: 110017K->110017K(1142784K)
[27.439s][info   ][gc             ] GC(17) Pause Young (Allocation Failure) 1868M->734M(2743M) 120.470ms
[27.439s][info   ][gc,cpu         ] GC(17) User=0.73s Sys=0.08s Real=0.12s
[27.439s][info   ][gc,start       ] GC(18) Pause Full (Ergonomics)
[27.439s][info   ][gc,phases,start] GC(18) Marking Phase
[27.829s][info   ][gc,phases      ] GC(18) Marking Phase 389.552ms
[27.829s][info   ][gc,phases,start] GC(18) Summary Phase
[27.829s][info   ][gc,phases      ] GC(18) Summary Phase 0.023ms
[27.829s][info   ][gc,phases,start] GC(18) Adjust Roots
[28.003s][info   ][gc,phases      ] GC(18) Adjust Roots 174.085ms
[28.003s][info   ][gc,phases,start] GC(18) Compaction Phase
[29.962s][info   ][gc,phases      ] GC(18) Compaction Phase 1958.726ms
[29.962s][info   ][gc,phases,start] GC(18) Post Compact
[29.964s][info   ][gc,phases      ] GC(18) Post Compact 2.096ms
[29.965s][info   ][gc,heap        ] GC(18) PSYoungGen: 73172K->32274K(2119680K)
[29.965s][info   ][gc,heap        ] GC(18) ParOldGen: 678829K->699349K(1288192K)
[29.965s][info   ][gc,metaspace   ] GC(18) Metaspace: 110017K->109985K(1142784K)
[29.965s][info   ][gc             ] GC(18) Pause Full (Ergonomics) 734M->714M(3328M) 2525.974ms
[29.965s][info   ][gc,cpu         ] GC(18) User=13.11s Sys=0.05s Real=2.52s
[35.201s][info   ][gc,start       ] GC(19) Pause Young (Allocation Failure)
[35.387s][info   ][gc,heap        ] GC(19) PSYoungGen: 1981970K->100832K(2129408K)
[35.387s][info   ][gc,heap        ] GC(19) ParOldGen: 699349K->712072K(1288192K)
[35.387s][info   ][gc,metaspace   ] GC(19) Metaspace: 111621K->111621K(1144832K)
[35.387s][info   ][gc             ] GC(19) Pause Young (Allocation Failure) 2618M->793M(3337M) 186.522ms
[35.387s][info   ][gc,cpu         ] GC(19) User=1.06s Sys=0.04s Real=0.18s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]   (typeflow):  18,649.87 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]    (objects):  13,697.69 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]   (features):     591.92 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]     analysis:  34,723.92 ms
[42.779s][info   ][gc,start       ] GC(20) Pause Young (Allocation Failure)
[42.963s][info   ][gc,heap        ] GC(20) PSYoungGen: 2050528K->147253K(2423296K)
[42.963s][info   ][gc,heap        ] GC(20) ParOldGen: 712072K->712080K(1288192K)
[42.963s][info   ][gc,metaspace   ] GC(20) Metaspace: 112173K->112173K(1144832K)
[42.963s][info   ][gc             ] GC(20) Pause Young (Allocation Failure) 2697M->839M(3624M) 184.024ms
[42.963s][info   ][gc,cpu         ] GC(20) User=0.85s Sys=0.05s Real=0.18s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]     (clinit):     716.27 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]     universe:   2,345.17 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]      (parse):   2,472.09 ms
[49.421s][info   ][gc,start       ] GC(21) Pause Young (Allocation Failure)
[49.762s][info   ][gc,heap        ] GC(21) PSYoungGen: 2392885K->192504K(2438144K)
[49.762s][info   ][gc,heap        ] GC(21) ParOldGen: 712080K->1133754K(1288192K)
[49.762s][info   ][gc,metaspace   ] GC(21) Metaspace: 113126K->113126K(1146880K)
[49.762s][info   ][gc             ] GC(21) Pause Young (Allocation Failure) 3032M->1295M(3639M) 340.895ms
[49.762s][info   ][gc,cpu         ] GC(21) User=2.07s Sys=0.24s Real=0.34s
[49.762s][info   ][gc,start       ] GC(22) Pause Full (Ergonomics)
[49.762s][info   ][gc,phases,start] GC(22) Marking Phase
[50.108s][info   ][gc,phases      ] GC(22) Marking Phase 345.921ms
[50.108s][info   ][gc,phases,start] GC(22) Summary Phase
[50.108s][info   ][gc,phases      ] GC(22) Summary Phase 0.046ms
[50.108s][info   ][gc,phases,start] GC(22) Adjust Roots
[50.188s][info   ][gc,phases      ] GC(22) Adjust Roots 79.838ms
[50.188s][info   ][gc,phases,start] GC(22) Compaction Phase
[51.219s][info   ][gc,phases      ] GC(22) Compaction Phase 1031.125ms
[51.219s][info   ][gc,phases,start] GC(22) Post Compact
[51.222s][info   ][gc,phases      ] GC(22) Post Compact 2.665ms
[51.223s][info   ][gc,heap        ] GC(22) PSYoungGen: 192504K->0K(2861568K)
[51.223s][info   ][gc,heap        ] GC(22) ParOldGen: 1133754K->677215K(1697792K)
[51.223s][info   ][gc,metaspace   ] GC(22) Metaspace: 113126K->113114K(1146880K)
[51.223s][info   ][gc             ] GC(22) Pause Full (Ergonomics) 1295M->661M(4452M) 1460.440ms
[51.223s][info   ][gc,cpu         ] GC(22) User=8.63s Sys=0.04s Real=1.46s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]     (inline):   5,954.11 ms
[55.667s][info   ][gc,start       ] GC(23) Pause Young (Allocation Failure)
[55.691s][info   ][gc,heap        ] GC(23) PSYoungGen: 2485248K->52897K(2861568K)
[55.691s][info   ][gc,heap        ] GC(23) ParOldGen: 677215K->677223K(1697792K)
[55.691s][info   ][gc,metaspace   ] GC(23) Metaspace: 116290K->116290K(1148928K)
[55.691s][info   ][gc             ] GC(23) Pause Young (Allocation Failure) 3088M->713M(4452M) 23.714ms
[55.691s][info   ][gc,cpu         ] GC(23) User=0.17s Sys=0.00s Real=0.03s
[60.693s][info   ][gc,start       ] GC(24) Pause Young (Allocation Failure)
[60.738s][info   ][gc,heap        ] GC(24) PSYoungGen: 2538145K->78459K(3010560K)
[60.738s][info   ][gc,heap        ] GC(24) ParOldGen: 677223K->677231K(1697792K)
[60.738s][info   ][gc,metaspace   ] GC(24) Metaspace: 118180K->118180K(1150976K)
[60.738s][info   ][gc             ] GC(24) Pause Young (Allocation Failure) 3140M->737M(4598M) 44.765ms
[60.738s][info   ][gc,cpu         ] GC(24) User=0.25s Sys=0.02s Real=0.04s
[65.003s][info   ][gc,start       ] GC(25) Pause Young (Allocation Failure)
[65.064s][info   ][gc,heap        ] GC(25) PSYoungGen: 2785915K->114616K(3061248K)
[65.064s][info   ][gc,heap        ] GC(25) ParOldGen: 677231K->677239K(1697792K)
[65.064s][info   ][gc,metaspace   ] GC(25) Metaspace: 118390K->118390K(1150976K)
[65.064s][info   ][gc             ] GC(25) Pause Young (Allocation Failure) 3381M->773M(4647M) 60.517ms
[65.064s][info   ][gc,cpu         ] GC(25) User=0.38s Sys=0.06s Real=0.06s
[68.794s][info   ][gc,start       ] GC(26) Pause Young (Allocation Failure)
[68.884s][info   ][gc,heap        ] GC(26) PSYoungGen: 2822072K->141632K(3231232K)
[68.884s][info   ][gc,heap        ] GC(26) ParOldGen: 677239K->677247K(1697792K)
[68.884s][info   ][gc,metaspace   ] GC(26) Metaspace: 118485K->118485K(1150976K)
[68.885s][info   ][gc             ] GC(26) Pause Young (Allocation Failure) 3417M->799M(4813M) 90.110ms
[68.885s][info   ][gc,cpu         ] GC(26) User=0.41s Sys=0.05s Real=0.09s
[72.556s][info   ][gc,start       ] GC(27) Pause Young (Allocation Failure)
[72.635s][info   ][gc,heap        ] GC(27) PSYoungGen: 3069248K->176772K(3245568K)
[72.635s][info   ][gc,heap        ] GC(27) ParOldGen: 677247K->677255K(1697792K)
[72.635s][info   ][gc,metaspace   ] GC(27) Metaspace: 118542K->118542K(1150976K)
[72.636s][info   ][gc             ] GC(27) Pause Young (Allocation Failure) 3658M->834M(4827M) 79.908ms
[72.636s][info   ][gc,cpu         ] GC(27) User=0.54s Sys=0.06s Real=0.08s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]    (compile):  22,243.07 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]      compile:  32,357.08 ms
[79.568s][info   ][gc,start       ] GC(28) Pause Young (Allocation Failure)
[79.681s][info   ][gc,heap        ] GC(28) PSYoungGen: 3104388K->262105K(3449344K)
[79.681s][info   ][gc,heap        ] GC(28) ParOldGen: 677255K->677263K(1697792K)
[79.681s][info   ][gc,metaspace   ] GC(28) Metaspace: 118975K->118975K(1153024K)
[79.681s][info   ][gc             ] GC(28) Pause Young (Allocation Failure) 3693M->917M(5026M) 113.435ms
[79.681s][info   ][gc,cpu         ] GC(28) User=0.66s Sys=0.10s Real=0.12s
[validation-quickstart-1.0-SNAPSHOT-runner:18832]        image:   3,162.48 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]        write:     440.05 ms
[validation-quickstart-1.0-SNAPSHOT-runner:18832]      [total]:  79,799.78 ms
[80.142s][info   ][gc,heap,exit   ] Heap
[80.142s][info   ][gc,heap,exit   ]  PSYoungGen      total 3449344K, used 336673K [0x00000006d5580000, 0x00000007bfd00000, 0x0000000800000000)
[80.142s][info   ][gc,heap,exit   ]   eden space 3131904K, 2% used [0x00000006d5580000,0x00000006d9e51fd8,0x0000000794800000)
[80.142s][info   ][gc,heap,exit   ]   from space 317440K, 82% used [0x00000007a9b00000,0x00000007b9af66a0,0x00000007bd100000)
[80.142s][info   ][gc,heap,exit   ]   to   space 347136K, 0% used [0x0000000794800000,0x0000000794800000,0x00000007a9b00000)
[80.142s][info   ][gc,heap,exit   ]  ParOldGen       total 1697792K, used 677263K [0x0000000480000000, 0x00000004e7a00000, 0x00000006d5580000)
[80.142s][info   ][gc,heap,exit   ]   object space 1697792K, 39% used [0x0000000480000000,0x00000004a9563f20,0x00000004e7a00000)
[80.142s][info   ][gc,heap,exit   ]  Metaspace       used 119119K, capacity 121934K, committed 122200K, reserved 1153024K
[80.142s][info   ][gc,heap,exit   ]   class space    used 18205K, capacity 19288K, committed 19328K, reserved 1048576K

Naturally the GC log format is different between 8 and 11, but I think we can see a similar number of GC events. But the final stats seem pretty divergent.

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 4, 2020

Here is a comparison of the final stats in tabular form for readability:

Item Java 8 Java 11
PSYoungGen total 2,745,344K 3,449,344K
PSYoungGen used 1,327,332K 336,673K
eden space 2,297,344K / 52% used 3,131,904K / 2% used
from space 448,000K / 26% used 317,440K / 82% used
to space 420,352K / 0% used 347,136K / 0% used
ParOldGen total 1,610,240K 1,697,792K
ParOldGen used 1,276,833K 677,263K
object space 1,610,240K / 79% used 1,697,792K / 39% used
Metaspace used 107,813K 119,119K
Metaspace capacity 113,351K 121,934K
Metaspace committed 113,664K 122,200K
Metaspace reserved 1,146,880K 1,153,024K
Class space used 15,908K 18,205K
Class space capacity 16,772K 19,288K
Class space committed 16,896K 19,328K
Class space reserved 1,048,576K 1,048,576K

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 4, 2020

Here's a detailed diff of the options being used as reported by -XX:+PrintFlagsFinal:

--- 8.filt.txt	2020-02-04 14:04:55.258981173 -0600
+++ 11.filt.txt	2020-02-04 14:04:50.427146608 -0600
@@ -2,18 +2,17 @@
 ActiveProcessorCount=-1
 AdaptiveSizeDecrementScaleFactor=4
 AdaptiveSizeMajorGCDecayTimeScale=10
-AdaptiveSizePausePolicy=0
 AdaptiveSizePolicyCollectionCostMargin=50
 AdaptiveSizePolicyInitializingSteps=20
 AdaptiveSizePolicyOutputInterval=0
 AdaptiveSizePolicyWeight=10
 AdaptiveSizeThroughPutPolicy=0
 AdaptiveTimeWeight=25
-AdjustConcurrency=false
 AggressiveHeap=false
 AggressiveOpts=false
 AliasLevel=3
 AlignVector=false
+AllocateHeapAt=
 AllocateInstancePrefetchLines=1
 AllocatePrefetchDistance=192
 AllocatePrefetchInstr=0
@@ -24,29 +23,28 @@
 AllowNonVirtualCalls=false
 AllowParallelDefineClass=false
 AllowUserSignalHandlers=false
+AllowVectorizeOnDemand=true
 AlwaysActAsServerClassMachine=false
 AlwaysCompileLoopMethods=false
 AlwaysLockClassLoader=false
 AlwaysPreTouch=false
 AlwaysRestoreFPU=false
 AlwaysTenure=false
+ArrayCopyLoadStoreMaxElem=8
 AssertOnSuspendWaitFailure=false
-AssumeMP=false
+AssumeMP=true
 AutoBoxCacheMax=128
-AutoGCSelectPauseMillis=5000
 BCEATraceLevel=0
-BackEdgeThreshold=100000
 BackgroundCompilation=true
 BaseFootPrintEstimate=268435456
 BiasedLockingBulkRebiasThreshold=20
 BiasedLockingBulkRevokeThreshold=40
 BiasedLockingDecayTime=25000
-BiasedLockingStartupDelay=4000
+BiasedLockingStartupDelay=0
 BindGCTaskThreadsToCPUs=false
 BlockLayoutByFrequency=true
 BlockLayoutMinDiamondPercentage=20
 BlockLayoutRotateLoops=true
-BootstrapJVMCI=false
 BranchOnRegister=false
 BytecodeVerificationLocal=false
 BytecodeVerificationRemote=true
@@ -57,7 +55,7 @@
 C1ProfileInlinedCalls=true
 C1ProfileVirtualCalls=true
 C1UpdateMethodData=true
-CICompilerCount:=4
+CICompilerCount=4
 CICompilerCountPerCPU=true
 CITime=false
 CMSAbortSemantics=false
@@ -68,20 +66,12 @@
 CMSClassUnloadingEnabled=true
 CMSClassUnloadingMaxInterval=0
 CMSCleanOnEnter=true
-CMSCompactWhenClearAllSoftRefs=true
 CMSConcMarkMultiple=32
 CMSConcurrentMTEnabled=true
 CMSCoordinatorYieldSleepCount=10
-CMSDumpAtPromotionFailure=false
 CMSEdenChunksRecordAlways=true
 CMSExpAvgFactor=50
 CMSExtrapolateSweep=false
-CMSFullGCsBeforeCompaction=0
-CMSIncrementalDutyCycle=10
-CMSIncrementalDutyCycleMin=0
-CMSIncrementalMode=false
-CMSIncrementalOffset=0
-CMSIncrementalPacing=true
 CMSIncrementalSafetyFactor=10
 CMSIndexedFreeListReplenish=4
 CMSInitiatingOccupancyFraction=-1
@@ -98,7 +88,6 @@
 CMSOldPLABResizeQuicker=false
 CMSOldPLABToleranceFactor=4
 CMSPLABRecordAlways=true
-CMSParPromoteBlocksToClaim=16
 CMSParallelInitialMarkEnabled=true
 CMSParallelRemarkEnabled=true
 CMSParallelSurvivorRemarkEnabled=true
@@ -112,7 +101,6 @@
 CMSPrecleanThreshold=1000
 CMSPrecleaningEnabled=true
 CMSPrintChunksInDump=false
-CMSPrintEdenSurvivorChunks=false
 CMSPrintObjectsInDump=false
 CMSRemarkVerifyVariant=1
 CMSReplenishIntermediate=true
@@ -137,48 +125,41 @@
 CMS_SweepPadding=1
 CMS_SweepTimerThresholdMillis=10
 CMS_SweepWeight=75
-CheckEndorsedAndExtDirs=false
+CalculateClassFingerprint=false
 CheckJNICalls=false
 ClassUnloading=true
 ClassUnloadingWithConcurrentMark=true
-ClearFPUAtPark=0
 ClipInlining=true
 CodeCacheExpansionSize=65536
-CodeCacheMinimumFreeSpace=512000
-CodeInstallSafepointChecks=true
-CollectGen0First=false
 CompactFields=true
-CompilationPolicyChoice=3
+CompactStrings=true
+CompilationPolicyChoice=2
 CompileCommand=
 CompileCommandFile=
 CompileOnly=
 CompileThreshold=10000
-CompilerThreadHintNoPreempt=true
+CompileThresholdScaling=1.000000
+CompilerThreadHintNoPreempt=false
 CompilerThreadPriority=-1
 CompilerThreadStackSize=2048
 CompressedClassSpaceSize=1073741824
 ConcGCThreads=0
 ConditionalMoveLimit=3
 ContendedPaddingWidth=128
-ConvertSleepToYield=true
-ConvertYieldToSleep=false
 CrashOnOutOfMemoryError=false
-CreateMinidumpOnCrash=false
+CreateCoredumpOnCrash=true
 CriticalJNINatives=true
 DTraceAllocProbes=false
 DTraceMethodProbes=false
 DTraceMonitorProbes=false
 Debugging=false
-DefaultMaxRAMFraction=4
-DefaultThreadPriority=-1
-DeferPollingPageLoopCount=-1
-DeferThrSuspendLoopCount=4000
 DeoptimizeRandom=false
 DisableAttachMechanism=false
 DisableExplicitGC=false
 DisplayVMOutputToStderr=false
 DisplayVMOutputToStdout=false
 DoEscapeAnalysis=true
+DoReserveCopyInSuperWord=true
 DontCompileHugeMethods=true
 DontYieldALot=false
 DumpLoadedClassList=
@@ -191,28 +172,30 @@
 EliminateAutoBox=true
 EliminateLocks=true
 EliminateNestedLocks=true
-EmitSync=0
 EnableContended=true
+EnableDynamicAgentLoading=true
 EnableJVMCI=true
-EnableTracing=false
+EnableJVMCIProduct=true
 ErgoHeapSizeLimit=0
 ErrorFile=
+ErrorLogTimeout=120
 ErrorReportServer=
 EscapeAnalysisTimeout=20.000000
 EstimateArgEscape=true
+ExecutingUnitTests=false
 ExitOnOutOfMemoryError=false
 ExplicitGCInvokesConcurrent=false
-ExplicitGCInvokesConcurrentAndUnloadsClasses=false
 ExtendedDTraceProbes=false
+ExtensiveErrorReports=false
 ExtraSharedClassListFile=
 FLSAlwaysCoalesceLarge=false
 FLSCoalescePolicy=2
 FLSLargestBlockCoalesceProximity=0.990000
 FailOverToOldVerifier=true
-FastTLABRefill=true
-FenceInstruction=0
 FieldsAllocationStyle=1
 FilterSpuriousWakeups=true
+FlightRecorder=false
+FlightRecorderOptions=
 ForceNUMA=false
 ForceTimeHighResolution=false
 FreqInlineSize=325
@@ -223,7 +206,7 @@
 G1ConcRefinementRedZone=0
 G1ConcRefinementServiceIntervalMillis=300
 G1ConcRefinementThreads=0
-G1ConcRefinementThresholdStep=0
+G1ConcRefinementThresholdStep=2
 G1ConcRefinementYellowZone=0
 G1ConfidencePercent=50
 G1HeapRegionSize=0
@@ -233,17 +216,17 @@
 G1RSetScanBlockSize=64
 G1RSetSparseRegionEntries=0
 G1RSetUpdatingPauseTimePercent=10
-G1RefProcDrainInterval=10
+G1RefProcDrainInterval=1000
 G1ReservePercent=10
 G1SATBBufferEnqueueingThresholdPercent=60
 G1SATBBufferSize=1024
 G1UpdateBufferSize=256
 G1UseAdaptiveConcRefinement=true
+G1UseAdaptiveIHOP=true
 GCDrainStackTargetSize=64
 GCHeapFreeLimit=2
 GCLockerEdenExpansionPercent=5
 GCLockerInvokesConcurrent=false
-GCLogFileSize=8192
 GCPauseIntervalMillis=0
 GCTaskTimeStampEntries=200
 GCTimeLimit=98
@@ -255,14 +238,15 @@
 HeapDumpPath=
 HeapFirstMaximumCompactionCount=3
 HeapMaximumCompactionInterval=20
-HeapSizePerGCThread=87241520
+HeapSearchSteps=3
+HeapSizePerGCThread=43620760
 IgnoreEmptyClassPaths=false
 IgnoreUnrecognizedVMOptions=false
 IncreaseFirstTierCompileThresholdAt=50
 IncrementalInline=true
 InitialBootClassLoaderMetaspaceSize=4194304
 InitialCodeCacheSize=2555904
-InitialHeapSize:=526385152
+InitialHeapSize=526385152
 InitialRAMFraction=64
 InitialRAMPercentage=1.562500
 InitialSurvivorRatio=8
@@ -275,16 +259,13 @@
 InsertMemBarAfterArraycopy=true
 InteriorEntryAlignment=16
 InterpreterProfilePercentage=33
-JNIDetachReleasesMonitors=true
 JVMCICounterSize=0
 JVMCICountersExcludeCompiler=true
-JVMCIHostThreads=1
 JVMCILibDumpJNIConfig=
 JVMCILibPath=
 JVMCINMethodSizeLimit=655360
 JVMCIPrintProperties=false
 JVMCIThreads=1
-JVMCITraceLevel=0
 JavaMonitorsInStackTrace=true
 JavaPriority10_To_OSPriority=-1
 JavaPriority1_To_OSPriority=-1
@@ -299,11 +280,13 @@
 LIRFillDelaySlots=false
 LargePageHeapSizeThreshold=134217728
 LargePageSizeInBytes=0
-LazyBootClassLoader=true
 LiveNodeCountInliningCutoff=40000
 LoadExecStackDllInVMThread=true
 LoopMaxUnroll=16
 LoopOptsCount=43
+LoopPercentProfileLimit=30
+LoopStripMiningIter=0
+LoopStripMiningIterShortLoop=0
 LoopUnrollLimit=60
 LoopUnrollMin=4
 LoopUnswitching=true
@@ -317,9 +300,9 @@
 MaxDirectMemorySize=0
 MaxFDLimit=true
 MaxGCMinorPauseMillis=18446744073709551615
-MaxGCPauseMillis=18446744073709551615
+MaxGCPauseMillis=18446744073709551614
 MaxHeapFreeRatio=100
-MaxHeapSize:=8405385216
+MaxHeapSize=8405385216
 MaxInlineLevel=9
 MaxInlineSize=35
 MaxJNILocalCapacity=65536
@@ -331,8 +314,8 @@
 MaxMetaspaceExpansion=5451776
 MaxMetaspaceFreeRatio=70
 MaxMetaspaceSize=18446744073709547520
-MaxNewSize:=2801795072
-MaxNodeLimit=75000
+MaxNewSize=2801795072
+MaxNodeLimit=80000
 MaxRAM=137438953472
 MaxRAMFraction=4
 MaxRAMPercentage=25.000000
@@ -342,8 +325,7 @@
 MaxVectorSize=32
 MetaspaceSize=21807104
 MethodFlushing=true
-MethodProfileWidth=0
-MinHeapDeltaBytes:=524288
+MinHeapDeltaBytes=524288
 MinHeapFreeRatio=0
 MinInliningThreshold=250
 MinJumpTableSize=10
@@ -354,9 +336,8 @@
 MinSurvivorRatio=3
 MinTLABSize=2048
 MonitorBound=0
-MonitorInUseLists=false
+MonitorInUseLists=true
 MultiArrayExpandLimit=6
-MustCallLoadClassInternal=false
 NUMAChunkResizeWeight=20
 NUMAInterleaveGranularity=2097152
 NUMAPageScanRate=256
@@ -367,18 +348,17 @@
 NeverActAsServerClassMachine=false
 NeverTenure=false
 NewRatio=2
-NewSize:=175112192
+NewSize=175112192
 NewSizeThreadIncrease=5320
 NmethodSweepActivity=10
-NmethodSweepCheckInterval=5
-NmethodSweepFraction=16
 NodeLimitFudgeFactor=2000
-NumberOfGCLogFiles=0
+NonNMethodCodeHeapSize=5836300
+NonProfiledCodeHeapSize=122910970
 NumberOfLoopInstrToAlign=4
 ObjectAlignmentInBytes=8
 OldPLABSize=1024
 OldPLABWeight=50
-OldSize:=351272960
+OldSize=351272960
 OmitStackTraceInFastThrow=true
 OnError=
 OnOutOfMemoryError=
@@ -388,6 +368,7 @@
 OptimizeStringConcat=true
 OptoBundling=false
 OptoLoopAlignment=16
+OptoRegScheduling=true
 OptoScheduling=false
 PLABWeight=75
 PSChunkLargeArrays=true
@@ -397,7 +378,6 @@
 ParGCUseLocalOverflow=false
 ParallelGCBufferWastePct=10
 ParallelGCThreads=8
-ParallelGCVerbose=false
 ParallelOldDeadWoodLimiterMean=50
 ParallelOldDeadWoodLimiterStdDev=80
 ParallelRefProcBalancingEnabled=true
@@ -418,7 +398,7 @@
 PerfDataSaveToFile=false
 PerfDisableSharedMem=false
 PerfMaxStringConstLength=1024
-PreInflateSpin=10
+PreTouchParallelChunkSize=1073741824
 PreferContainerQuotaForCPUCount=true
 PreferInterpreterNativeStubs=false
 PrefetchCopyIntervalInBytes=576
@@ -427,54 +407,26 @@
 PreserveAllAnnotations=false
 PreserveFramePointer=false
 PretenureSizeThreshold=0
-PrintAdaptiveSizePolicy=false
-PrintBootstrap=true
-PrintCMSInitiationStatistics=false
-PrintCMSStatistics=0
 PrintClassHistogram=false
-PrintClassHistogramAfterFullGC=false
-PrintClassHistogramBeforeFullGC=false
 PrintCodeCache=false
 PrintCodeCacheOnCompilation=false
 PrintCommandLineFlags=false
 PrintCompilation=false
 PrintConcurrentLocks=false
-PrintDeoptimizationDetails=false
-PrintFLSCensus=0
-PrintFLSStatistics=0
-PrintFlagsFinal:=true
+PrintExtendedThreadInfo=false
+PrintFlagsFinal=true
 PrintFlagsInitial=false
+PrintFlagsRanges=false
 PrintGC=false
-PrintGCApplicationConcurrentTime=false
-PrintGCApplicationStoppedTime=false
-PrintGCCause=true
-PrintGCDateStamps=false
 PrintGCDetails=false
-PrintGCID=false
-PrintGCTaskTimeStamps=false
-PrintGCTimeStamps=false
-PrintHeapAtGC=false
-PrintHeapAtGCExtended=false
 PrintHeapAtSIGBREAK=true
-PrintJNIGCStalls=false
 PrintJNIResolving=false
-PrintNMethodStatistics=false
-PrintOldPLAB=false
-PrintOopAddress=false
-PrintPLAB=false
-PrintParallelOldGCPhaseTimes=false
-PrintPromotionFailure=false
-PrintReferenceGC=false
 PrintSafepointStatistics=false
 PrintSafepointStatisticsCount=300
 PrintSafepointStatisticsTimeout=-1
 PrintSharedArchiveAndExit=false
 PrintSharedDictionary=false
-PrintSharedSpaces=false
-PrintStringDeduplicationStatistics=false
 PrintStringTableStatistics=false
-PrintTLAB=false
-PrintTenuringDistribution=false
 PrintTieredEvents=false
 PrintVMOptions=false
 PrintVMQWaitTime=false
@@ -485,20 +437,19 @@
 ProfileIntervalsTicks=100
 ProfileMaturityPercentage=20
 ProfileVM=false
+ProfiledCodeHeapSize=122910970
 ProfilerPrintByteCodeStatistics=false
 ProfilerRecordPC=false
 PromotedPadding=3
 QueuedAllocationWarningCount=0
 RTMRetryCount=5
 RangeCheckElimination=true
-ReadPrefetchInstr=0
 ReassociateInvariants=true
 ReduceBulkZeroing=true
 ReduceFieldZeroing=true
 ReduceInitialCardMarks=true
 ReduceSignalUsage=false
 RefDiscoveryPolicy=0
-ReflectionWrapResolutionErrors=true
 RegisterFinalizersAtInit=true
 RelaxAccessControlCheck=false
 ReplayDataFile=
@@ -509,42 +460,41 @@
 ResizeTLAB=true
 RestoreMXCSROnJNICalls=false
 RestrictContended=true
+RestrictReservedStack=true
 RewriteBytecodes=true
 RewriteFrequentPairs=true
-SafepointPollOffset=256
-SafepointSpinBeforeYield=2000
 SafepointTimeout=false
 SafepointTimeoutDelay=10000
 ScavengeBeforeFullGC=true
+SegmentedCodeCache=true
 SelfDestructTimer=0
-ShareDebugInfo=true
+SharedArchiveConfigFile=
+SharedArchiveFile=
 SharedBaseAddress=34359738368
 SharedClassListFile=
-SharedMiscCodeSize=122880
-SharedMiscDataSize=4194304
-SharedReadOnlySize=16777216
-SharedReadWriteSize=16777216
+SharedSymbolTableBucketSize=4
 ShowMessageBoxOnError=false
+ShrinkHeapInSteps=true
 SoftRefLRUPolicyMSPerMB=1000
-SpecialEncodeISOArray=true
 SplitIfBlocks=true
 StackRedPages=1
+StackReservedPages=1
 StackShadowPages=20
 StackTraceInThrowable=true
 StackYellowPages=2
+StartAggressiveSweepingAt=10
 StartAttachListener=false
-StarvationMonitorInterval=200
+StartFlightRecording=
 StressLdcRewrite=false
 StringDeduplicationAgeThreshold=3
-StringTableSize=60013
+StringTableSize=65536
+SuperWordLoopUnrollAnalysis=true
+SuperWordReductions=true
 SuppressFatalErrorMessage=false
 SurvivorPadding=3
 SurvivorRatio=8
 SuspendRetryCount=50
 SuspendRetryDelay=5
-SyncFlags=0
-SyncKnobs=
-SyncVerbose=0
 TLABAllocationWeight=35
 TLABRefillWasteFraction=64
 TLABSize=0
@@ -556,9 +506,9 @@
 TenuredGenerationSizeIncrement=20
 TenuredGenerationSizeSupplement=80
 TenuredGenerationSizeSupplementDecay=2
+ThreadLocalHandshakes=true
 ThreadPriorityPolicy=0
 ThreadPriorityVerbose=false
-ThreadSafetyMargin=52428800
 ThreadStackSize=1024
 ThresholdTolerance=10
 Tier0BackedgeNotifyFreqLog=10
@@ -569,6 +519,10 @@
 Tier2BackedgeNotifyFreqLog=14
 Tier2CompileThreshold=0
 Tier2InvokeNotifyFreqLog=11
+Tier3AOTBackEdgeThreshold=120000
+Tier3AOTCompileThreshold=15000
+Tier3AOTInvocationThreshold=10000
+Tier3AOTMinInvocationThreshold=1000
 Tier3BackEdgeThreshold=60000
 Tier3BackedgeNotifyFreqLog=13
 Tier3CompileThreshold=2000
@@ -589,39 +543,20 @@
 TieredRateUpdateMinTime=1
 TieredStopAtLevel=4
 TimeLinearScan=false
-TraceBiasedLocking=false
-TraceClassLoading=false
-TraceClassLoadingCause=
-TraceClassLoadingPreorder=false
-TraceClassPaths=false
-TraceClassResolution=false
-TraceClassUnloading=false
-TraceDeoptimization=false
-TraceDynamicGCThreads=false
-TraceGen0Time=false
-TraceGen1Time=false
 TraceJVMTI=
-TraceLoaderConstraints=false
-TraceMetadataHumongousAllocation=false
-TraceMonitorInflation=false
-TraceParallelOldGCTasks=false
-TraceRedefineClasses=0
-TraceSafepointCleanupTime=false
 TraceSuspendWaitFailures=false
 TrackedInitializationLimit=50
 TransmitErrorReport=false
 TrapBasedNullChecks=false
 TrapBasedRangeChecks=false
 TypeProfileArgsLimit=2
-TypeProfileLevel=111
+TypeProfileLevel=0
 TypeProfileMajorReceiverPercent=90
 TypeProfileParmsLimit=2
 TypeProfileWidth=8
 UnguardOnExecutionViolation=0
 UnlinkSymbolsALot=false
-Use486InstrsOnly=false
 UseAES=true
-UseAESIntrinsics=true
 UseAVX=2
 UseAdaptiveGCBoundary=false
 UseAdaptiveGenerationSizePolicyAtMajorCollection=true
@@ -632,24 +567,20 @@
 UseAdaptiveSizePolicyFootprintGoal=true
 UseAdaptiveSizePolicyWithSystemGC=false
 UseAddressNop=true
-UseAltSigs=false
-UseAutoGCSelectPolicy=false
+UseBASE64Intrinsics=false
 UseBMI1Instructions=true
 UseBMI2Instructions=true
 UseBiasedLocking=true
 UseBimorphicInlining=true
-UseBoundThreads=true
 UseCLMUL=true
 UseCMSBestFit=true
-UseCMSCollectionPassing=true
-UseCMSCompactAtFullCollection=true
 UseCMSInitiatingOccupancyOnly=false
-UseCRC32Intrinsics=true
+UseCMoveUnconditionally=false
+UseCodeAging=true
 UseCodeCacheFlushing=true
 UseCompiler=true
-UseCompilerSafepoints=true
-UseCompressedClassPointers:=true
-UseCompressedOops:=true
+UseCompressedClassPointers=true
+UseCompressedOops=true
 UseConcMarkSweepGC=false
 UseCondCardMark=false
 UseContainerSupport=true
@@ -658,22 +589,19 @@
 UseCountedLoopSafepoints=false
 UseCounterDecay=true
 UseDivMod=true
-UseDynamicNumberOfGCThreads=false
+UseDynamicNumberOfCompilerThreads=true
+UseDynamicNumberOfGCThreads=true
+UseFMA=true
 UseFPUForSpilling=true
-UseFastAccessorMethods=false
-UseFastEmptyMethods=false
 UseFastJNIAccessors=true
 UseFastStosb=true
 UseG1GC=false
-UseGCLogFileRotation=false
 UseGCOverheadLimit=true
 UseGCTaskAffinity=false
-UseGHASHIntrinsics=true
 UseHeavyMonitors=false
 UseHugeTLBFS=false
 UseInlineCaches=true
 UseInterpreter=true
-UseJVMCIClassLoader=true
 UseJVMCICompiler=true
 UseJVMCINativeLibrary=true
 UseJumpTables=true
@@ -682,48 +610,38 @@
 UseLargePagesInMetaspace=false
 UseLargePagesIndividualAllocation=false
 UseLinuxPosixThreadCPUClocks=true
-UseLockedTracing=false
 UseLoopCounter=true
 UseLoopInvariantCodeMotion=true
 UseLoopPredicate=true
-UseMathExactIntrinsics=true
 UseMaximumCompactionOnSystemGC=true
-UseMembar=false
-UseMontgomeryMultiplyIntrinsic=true
-UseMontgomerySquareIntrinsic=true
-UseMulAddIntrinsic=true
-UseMultiplyToLenIntrinsic=true
+UseMembar=true
 UseNUMA=false
 UseNUMAInterleaving=false
 UseNewLongLShift=false
 UseOSErrorReporting=false
-UseOldInlining=true
 UseOnStackReplacement=true
 UseOnlyInlinedBimorphic=true
 UseOprofile=false
 UseOptoBiasInlining=true
 UsePSAdaptiveSurvivorSizePolicy=true
-UseParNewGC=false
-UseParallelGC:=true
+UseParallelGC=true
 UseParallelOldGC=true
 UsePerfData=true
 UsePopCountInstruction=true
+UseProfiledLoopPredicate=true
 UseRDPCForConstantTableBase=false
 UseRTMDeopt=false
 UseRTMLocking=false
-UseSHA=false
-UseSHA1Intrinsics=false
-UseSHA256Intrinsics=false
-UseSHA512Intrinsics=false
+UseSHA=true
 UseSHM=false
 UseSSE=4
 UseSSE42Intrinsics=true
 UseSerialGC=false
-UseSharedSpaces=false
+UseSharedSpaces=true
 UseSignalChaining=true
-UseSquareToLenIntrinsic=true
 UseStoreImmI16=false
 UseStringDeduplication=false
+UseSubwordForMaxVector=true
 UseSuperWord=true
 UseTLAB=true
 UseThreadPriorities=true
@@ -731,8 +649,9 @@
 UseTypeProfile=true
 UseTypeSpeculation=true
 UseUnalignedLoadStores=true
-UseVMInterruptibleIO=false
+UseVectorCmov=false
 UseXMMForArrayCopy=true
+UseXMMForObjInit=false
 UseXmmI2D=false
 UseXmmI2F=false
 UseXmmLoadAndClearUpper=true
@@ -745,11 +664,15 @@
 ValueSearchLimit=1000
 VerifyMergedCPBytecodes=true
 VerifySharedSpaces=false
-WorkAroundNPTLTimedWaitHang=1
 YoungGenerationSizeIncrement=20
 YoungGenerationSizeSupplement=80
 YoungGenerationSizeSupplementDecay=8
 YoungPLABSize=4096
-ZapTLAB=false
+ZAllocationSpikeTolerance=2.000000
+ZCollectionInterval=0
+ZFragmentationLimit=25.000000
+ZMarkStacksMax=8589934592
+ZPath=
+ZStallOnOutOfMemory=true
+ZStatisticsInterval=10
 ZeroTLAB=false

@pejovica
Copy link
Member

pejovica commented Feb 6, 2020

@pejovica We talked about switching the default GC to the parallel GC on JDK 11, but I think that is not done yet?

@christianwimmer Yes, it's not done yet.

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 6, 2020

@pejovica We talked about switching the default GC to the parallel GC on JDK 11, but I think that is not done yet?

@christianwimmer Yes, it's not done yet.

Setting it manually didn't make any significant difference. Was there some other tuning involved as well?

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 6, 2020

Breaking down the times a bit:

Java 8
Java 11

You can see that the initial setup times are actually a little bit better on Java 11. It's only once analysis begins that things take longer. So I guess the next step is to see whether there is a substantially larger number of classes/methods being processed under Java 11.

@SergejIsbrecht
Copy link

Maybe create an cpu flamegraph via async-profiler? I will try to do so with the agent tomorrow.

@jaikiran
Copy link
Contributor

jaikiran commented Feb 7, 2020

I remember seeing a similar issue a couple of months back when I was looking into something related to Quarkus with 19.3.0 when it was initially released. At that time my initial investigations had seemed to point to the "classpath"/classes that the com.oracle.svm.hosted.ImageClassLoader processes, was (unexpectedly?) larger when the Java 11 variant of 19.3.x as compared to Java 8 variant of 19.3.x. I didn't investigate it at that time because no one else seemed to run into it.

Now seeing this, I decided to dig a bit deeper. I used Byteman[1] to debug this issue. Since I had an idea that there's something different in ImageClassLoader, I used it as a starting point for forming a byteman rule. So the ImageClassLoader has a call to initAllClasses[2] which loads all the classes that are there in the classpath. That call ultimately lands into handleClassFileName[3] which is where the classes get loaded. So I added a byteman rule to print out to a file the classes that get loaded:

RULE native image init all classes
CLASS com.oracle.svm.hosted.ImageClassLoader
METHOD handleClassFileName
AT LINE 284
IF TRUE
DO traceOpen("logfile", "/tmp/class-list.txt");
   traceln("logfile", $className);
ENDRULE

All it does is prints out the className value into a file.

I ran this rule against the validator-quickstart that David mentioned in the first post of this issue:

mvn clean install -DskipTests=true -Dnative -Dquarkus.version=999-SNAPSHOT  -Dquarkus.native.additional-build-args="-J-javaagent:/home/me/byteman/byteman-4.0.7/lib/byteman.jar=script:/home/me/graalvm.btm\,boot:/home/me/byteman/byteman-4.0.7/lib/byteman.jar\,prop:org.jboss.byteman.verbose=true\,prop:org.jboss.byteman.transform.all=true"

I ran this with both the Java 8 and Java 11 variant of 19.3.1 of Graal VM. In the generated file (which is recreated before each run), for Java 8 variant it had 15928 classes and for the Java 11 variant it had 17372 classes. These numbers are constant across runs with both these variants. So Java 11 is loading 1444 additional classes as compared to the Java 8 version.
Now that I had files containing the class names, I wrote a quick trivial program to show which were the additionally classes that Java 11 was loading. It turns out, almost all (if not all) of them belong to com.oracle.truffle... packages.
I looked at the code in ImageClassLoader in detail to track down these classes and turns out there's this code[4] which specifically runs for Java version > 8 and loads a GRAALVM_HOME/lib/native-image-modules.list file and then loads all the classes from the modules listed in that file. The native-image-modules.list file (understandably only) exists only in the Java 11 variant of Graal VM and contains:

# Modules whose classes might have annotations processed by native-image
org.graalvm.sdk
org.graalvm.truffle
jdk.internal.vm.compiler
org.graalvm.locator
jdk.internal.vm.compiler.management
org.graalvm.js.scriptengine
jdk.aot
com.oracle.graal.graal_enterprise

which explains why it pulls in those additional 1444 classes from com.oracle.truffle package.

For reference, I've attached a graalvm-classlist.zip file to this comment and that zip contains a couple of files. They were generated by that Byteman script and contain the classes that are loaded, during Java 8 and Java 11 usage.

[1] https://byteman.jboss.org/
[2] https://github.com/oracle/graal/blob/release/graal-vm/19.3/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ImageClassLoader.java#L126
[3] https://github.com/oracle/graal/blob/release/graal-vm/19.3/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ImageClassLoader.java#L275.
[4] https://github.com/oracle/graal/blob/release/graal-vm/19.3/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ImageClassLoader.java#L141
[5] https://github.com/oracle/graal/blob/release/graal-vm/19.3/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ImageClassLoader.java#L156
graalvm-classlist.zip

@adinn
Copy link
Collaborator

adinn commented Feb 7, 2020

Nice detective work, Jaikiran. That Byteman thing sounds very useful ;-)

@jaikiran
Copy link
Contributor

jaikiran commented Feb 7, 2020

That Byteman thing sounds very useful ;-)

Indeed :)

@jaikiran
Copy link
Contributor

jaikiran commented Feb 7, 2020

So I tweaked that byteman rule a bit to print out the count of the applicationClasses, hostedOnlyClasses, systemMethods and systemFields of the ImageClassLoader[1], at the end of the ImageClassLoader instance creation:

RULE native image init all classes
CLASS com.oracle.svm.hosted.ImageClassLoader
METHOD create
AT EXIT
IF TRUE
DO traceOpen("logfile", "/tmp/class-list.txt");
   traceln("logfile", "Application classes " + $result.applicationClasses.size());
   traceln("logfile", "Hosted only classes " + $result.hostedOnlyClasses.size());
   traceln("logfile", "System methods " + $result.systemMethods.size());
   traceln("logfile", "System fields " + $result.systemFields.size());
ENDRULE

For Java 8 the results are:

Application classes 14591
Hosted only classes 610
System methods 96072
System fields 39653

For Java 11 the results are:

Application classes 16025
Hosted only classes 610
System methods 105200
System fields 42922

[1] https://github.com/oracle/graal/blob/release/graal-vm/19.3/substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/ImageClassLoader.java#L92

@dmlloyd
Copy link
Contributor Author

dmlloyd commented Feb 7, 2020

Great @jaikiran, thanks, and nice job with the analysis!

@lburgazzoli
Copy link

Not strictly related to this issue but adding truffle to the mix make some build fail on environments whit limited memory such as GitHub Actions ( ~ 7 GB ), see: https://gist.github.com/lburgazzoli/6dc563f6a1d76fc815a8297e6607735f

@cstancu
Copy link
Member

cstancu commented Apr 23, 2020

@jaikiran thanks for the in-depth analysis. I will look into how we can avoid the Truffle classes being pulled in. In the meantime, we added a feature that should improve analysis performance, see #2384.

@cstancu
Copy link
Member

cstancu commented Apr 23, 2020

So Java 11 is loading 1444 additional classes as compared to the Java 8 version.

@jaikiran Just because the extra classes from com.oracle.truffle.api are loaded by the ImageClassLoader it shouldn't make a significant difference in image building time, as long as that code is not actually reachable. (You can test that easily by removing the org.graalvm.truffle module declaration from ${GRAALVM/lib/native-image-modules.list.)

@olpaw why do we add ${GRAALVM}/lib/truffle/truffle-api.jar on the --module-path by default? Running native-image with --verbose for a simple HelloWorld app I see:

...
--module-path \
./projects/graal-ee-ws-0/graal/sdk/mxbuild/linux-amd64/GRAALVM_AA1C30D215_JAVA11/graalvm-aa1c30d215-java11-20.2.0-dev/lib/truffle/truffle-api.jar \
...

That combined with the org.graalvm.truffle module declaration in ${GRAALVM/lib/native-image-modules.list leads to loading all the com.oracle.truffle.api even when Truffle is not used.
Cleanin up the module path should be enough to avoid loading those extra classes.

@olpaw
Copy link
Member

olpaw commented Apr 30, 2020

That combined with the org.graalvm.truffle module declaration in ${GRAALVM/lib/native-image-modules.list leads to loading all the com.oracle.truffle.api even when Truffle is not used.

@cstancu we should fix that. Only if truffle-api.jar is actually needed we should have it on the module path. I'm currently working on several improvements in our handling of -imagecp and NativeImageClassLoader. Taking care of this truffle-api.jar problem fits well in. I will add a fix for that on PR #2415

@olpaw
Copy link
Member

olpaw commented Apr 30, 2020

@cstancu having truffle-api.jar on the --module-path is not a problem (since it's the module-path of the image-builder and not the module-path of the image we want to build). The problem arises from

Path list = javaHome.resolve(Paths.get("lib", "native-image-modules.list"));
if (Files.exists(list)) {
try {
Files.readAllLines(list).stream().map(s -> s.trim()).filter(s -> !s.isEmpty() && !s.startsWith("#")).forEach(modules::add);
} catch (IOException e) {
throw shouldNotReachHere(e);
}
}
for (String moduleResource : ModuleSupport.getModuleResources(modules)) {
if (moduleResource.endsWith(CLASS_EXTENSION)) {
executor.execute(() -> handleClassFileName(moduleResource, '/'));
}
}

This code unconditionally loads the classes from all modules in native-image-modules.list with NativeImageClassLoader. I'm currently testing if 4a5f6ae is a way to improve the situation. (I.e. only when truffles native-image.properties file is active we have -Dtruffle.TruffleRuntime=com.oracle.svm.truffle.api.SubstrateTruffleRuntime. Thus we should be able to use it as an indicator for building truffle images. cc @christianwimmer

@cstancu
Copy link
Member

cstancu commented Apr 30, 2020

This code unconditionally loads the classes from all modules in native-image-modules.list with NativeImageClassLoader

Yes, but if the truffle-api.jar is not on the --module-path then even if native-image-modules.list contains the org.graalvm.truffle declaration nothing extra will be loaded. In general, I think the best approach is to have a better decoupling of Truffle things.

@galderz
Copy link
Contributor

galderz commented Mar 22, 2021

Last week I had a look to see how different build times are. Running David's native image test in the description in my local environment, GraalVM CE 21.0.0.2 Java 11 is ~8-9% slower compared to the Java 8 version.

As @cstancu said on this comment, the fact that com.oracle.truffle.api classes are loaded by the ImageClassLoader does not have a noticeable impact on the build times (that happens in the Classes phase). The differences between Java 11 and Java 8 come in the Analysis and Image phases with the former being the biggest driver in increase.

Looking at the reports coming out of analysis (e.g. printed when passing in -H:+PrintAnalysisCallTree), the Java 11 version has 9% more reachable classes than Java 8. From what I could see, this seems mostly due to Java 11 itself, e.g. ASM classes used by Java 9+ module system, or SecureRandom implementation being more pluggable in more recent Java versions.

I didn't dig into the differences in Image phase, because although the increase is noticeable, it's quite small compared to the increase in the Analysis phase.

@cstancu
Copy link
Member

cstancu commented Dec 6, 2023

Native Image build times have been improved since this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

10 participants