ZoneRules.lastRuleCache side effects during source cache build fail native image generation

[galderz]

This is a bug in native image debug symbol generation. In particular, it's an issue with the way source cache generation is handled.

When ImageClassLoader walks the jars to find the classes, it inadvertently fills up the ZoneRules.lastRulesCache with time zone offset transitions for the years for which the classes were last modified. Typically, this would be filled up with years 2020 and 2019. See a typical stacktrace of this.

After the heap is creating, debug symbols generates the source cache, and at that port it can lookup classes in *-sources.jar files. This lookup can end up touching the same ZoneRules.lastRuleCache (see stacktrace), and if by chance, it adds a year that was not present after looking up classes, e.g. 2014, it will pollute the heap after snapshotting the heap. Native image detects this and fails with an error like:

[sourcecache-client/target/sourcecache:6632]        image:   6,911.30 ms,  3.25 GB
Fatal error:com.oracle.svm.core.util.VMError$HostedError: com.oracle.svm.core.util.UserError$UserException: Static field or an object referenced from a static field changed during native image generation?
  object:2017=[Ljava.time.zone.ZoneOffsetTransition;@5944088  of class: java.util.concurrent.ConcurrentHashMap$Node
  reachable through:
    object: [Ljava.util.concurrent.ConcurrentHashMap$Node;@ed99e14  of class: java.util.concurrent.ConcurrentHashMap$Node[]
    object: {2016=[Ljava.time.zone.ZoneOffsetTransition;@4194c6d5, 2017=[Ljava.time.zone.ZoneOffsetTransition;@5944088, 2019=[Ljava.time.zone.ZoneOffsetTransition;@118349c1, 2020=[Ljava.time.zone.ZoneOffsetTransition;@18a9bfe8, 2014=[Ljava.time.zone.ZoneOffsetTransition;@3258aabf, 2015=[Ljava.time.zone.ZoneOffsetTransition;@c4123b3}  of class: java.util.concurrent.ConcurrentHashMap
    object: ZoneRules[currentStandardOffset=Z]  of class: java.time.zone.ZoneRules
    object: Europe/London  of class: java.time.ZoneRegion
    object: sun.util.calendar.ZoneInfo[id="Europe/London",offset=0,dstSavings=3600000,useDaylight=true,transitions=242,lastRule=java.util.SimpleTimeZone[id=Europe/London,offset=0,dstSavings=3600000,useDaylight=true,startYear=0,startMode=2,startMonth=2,startDay=-1,startDayOfWeek=1,startTime=3600000,startTimeMode=2,endMode=2,endMonth=9,endDay=-1,endDayOfWeek=1,endTime=3600000,endTimeMode=2]]  of class: sun.util.calendar.ZoneInfo
    object: 0000-12-30T16:47:04.192Z  of class: sun.util.calendar.ImmutableGregorianDate
    object: [CommonEra (C.E.) since 0000-12-30T16:47:04.192 local time]  of class: sun.util.calendar.Era
    object: [Lsun.util.calendar.Era;@3bb95304  of class: sun.util.calendar.Era[]
    root: sun.util.calendar.JulianCalendar$Date.setNormalizedYear(int)

	at com.oracle.svm.core.util.VMError.shouldNotReachHere(VMError.java:72)
	at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:647)
	at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:451)
	at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1407)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)

I've built a reproducer for this issue here.

Some observations:

• Files.isRegularFile() cannot be substituted by File.isDirectory because the Path is a Path within a Zip FileSystem (e.g. a jar), and method is unsupported for that type of FS.
• This is a niche scenario, but the JDK tries to read all attributes (including time based ones), even when you only want to check if a file is a directory or not. Discussed with @adinn and this is fine.
• I've tried to do substitutions of zipfs.ZipUtils but obviously those don't have an effect during the heap build time.

I'm unsure yet about a fix here:

• One possibility would be to eagerly fill the ZoneUtils.lastRuleCache for the last N years, before snapshotting the heap. There's only one TZ in play here, which is ZoneId.systemDefault().
• Another option would be to use some reflection magic to clear lastRulesCache after navigating the classes, and clear it again after navigating the source cache (before writing the heap).

[galderz]

This was originally reported in quarkusio/quarkus#10309.

[galderz]

I'm testing out to see if recompute field value would work here, but I don't think it will. Even if you replace the value in the heap before computing the heap contents, you'd get rid of the mutations caused by the class file navigation, but the source cache would still mutate it and lead to the same error. You'd need a second recompute call before writing the heap (or some reflection magic).

[galderz]

Actually, recomputing the cache to be a new instance seems to work.