GraalVM Native Image Requires Config for lazy vals #13985
Comments
|
Long standing issue that should have received more attention. |
|
Some more context here: https://www.reddit.com/r/scala/comments/rzvkmi/graalvm_nativeimage_with_scala_3_reflection_hell/ |
|
hello, i tried with scala 3.2.0-RC3 and it's still happening. I don't see what release that one is supposed to be a part of, but it was checked in much before that date, is there any way I can confirm if it's in by that one or not? |
|
This fixing commit is present since |
|
Maybe need re-compile with new Scala 3 version all libraries 🤔 |
|
I bumped the |
|
As @jamesward example, I have no libraries, it's happening from just having a lazy in my own code |
|
Let's re-open the issue then, maybe @romanowski could have a look since he implemented the original fix in #14780 ? |
|
It still breaks because we still use reflection to acquire Unsafe instance. It does not need the classOf[sun.misc.Unsafe].getDeclaredFields.nn.find { field =>
field.nn.getType == classOf[sun.misc.Unsafe] && {
field.nn.setAccessible(true)
true
}
}If we accessed the Unsafe field |
|
IMO the best solution is to stop using sun.misc.Unsafe on Java 9+, i.e. fixing #9013. This will fix GraalVM problems because it's only Java 11 and Java 17. |
|
That would help too, but it won't completely solve the issue until Scala libraries that use lazy vals stop supporting Java 8. |
|
If we can't take the decision at runtime (java 8 vs java 9+) then yes :( |
|
I have a sample project using ZIO and Scala 3 and this issue happens. Running the Native Image binary gave me: I tried @jamesward workaround by creating [
{
"name":"sun.misc.Unsafe",
"allDeclaredFields":true
},
{
"name":"com.carlosedp.zioscalajs.backend.MainApp$package$",
"fields":[{"name":"0bitmap$1"}]
}
]and then had the error: Also tried removing the |
|
Tried this with latest 3.2.2-RC1 (which merged #15296) by enabling Also tested @jamesward sample repo and branch and same error happens. Do I need something else? |
|
It seems there's still something missing, @szymon-rd in #13985 (comment) suggested we need to go through |
|
@carlosedp |
|
Thanks @szymon-rd I'm out today on local holy day returning tomorrow afternoon. Gonna test as soon as I get in computer. |
|
Hey @szymon-rd , thanks a lot for the PR. I tried to go as deep as possible on this since the sample from above (simple LazyVal) worked but my sample app in Zio didn't. I went on adding the unsafe fields to One thing in common is that all of them refers to the Below the reflect-config: Click me[
{
"name": "izumi.reflect.macrortti.LightTypeTag",
"fields": [
{
"name": "0bitmap$1",
"allowUnsafeAccess": true
}
]
},
{
"name": "izumi.reflect.macrortti.LightTypeTag$ParsedLightTypeTag210",
"fields": [
{
"name": "0bitmap$5",
"allowUnsafeAccess": true
}
]
},
{
"name": "zhttp.http.Method$PUT$",
"fields": [
{
"name": "0bitmap$6",
"allowUnsafeAccess": true
}
]
},
{
"name": "zhttp.http.Method$DELETE$",
"fields": [
{
"name": "0bitmap$8",
"allowUnsafeAccess": true
}
]
},
{
"name": "zhttp.http.Method$POST$",
"fields": [
{
"name": "0bitmap$5",
"allowUnsafeAccess": true
}
]
},
{
"name": "zhttp.http.Method$GET$",
"fields": [
{
"name": "0bitmap$3",
"allowUnsafeAccess": true
}
]
},
{
"name": "izumi.reflect.macrortti.LightTypeTagRef$NameReference",
"fields": [
{
"name": "0bitmap$5",
"allowUnsafeAccess": true
}
]
},
{
"name": "izumi.reflect.macrortti.LightTypeTagInheritance",
"fields": [
{
"name": "0bitmap$1",
"allowUnsafeAccess": true
}
]
},
{
"name": "izumi.reflect.internal.fundamentals.platform.console.TrivialLogger$Config",
"fields": [
{
"name": "0bitmap$1",
"allowUnsafeAccess": true
}
]
}
]And the errors I found each execution: Click meThe project I've been testing on is at https://github.com/carlosedp/zio-scalajs-stack and it's configured to build with GraalVM and Scala 2.13 here https://github.com/carlosedp/zio-scalajs-stack/blob/main/build.sc#L57. I build with Let me know if I can help by providing any additional info. |
|
@carlosedp I think it is because these libraries were not compiled with the newer lazy vals. My PR should fix initialization of LazyVals runtime class. However, I don't think that we are able to change the runtime to make the libraries compiled with older Scala run without these exceptions. The bytecode of the libraries contains code that cannot be run without config. There are some solutions that can be implemented to make it work (any of these steps should help):
|
|
Thanks for the feedback Szymon. I don't think I have the knowledge to attempt solutions 2 and 3 so I'll try to test building the libs locally with new compiler and wait until they get published to have it upstream. Glad to know it should work from now on. Thanks a lot! |
|
I can confirm it works for my small Scala 3 + JavaFX project. Thanks! <3 |
|
An FYI, scala-cli does one of the options mentioned by @szymon-rd. It manipulates the bytecode and patches the jars to "fix" lazyvals on current libraries. |
|
I'm happy to add that since using Scala 3.3 (3.3.0-RC2), my ZIO project works perfectly with NativeImage. |
|
@carlosedp 3.3.0-RC2 has a bug for GraalVM, it sadly occurs fairly commonly and crashes the whole image. We will be backporting the fix for it: #16800 |
|
Thanks @szymon-rd! I've verified this fix: https://github.com/jamesward/scala3-graalvm-tester/tree/lazy-reflect-fixed |
Compiler version
3.1.0Background
GraalVM Native Image Ahead-of-Time compiles JVM apps into native executables. To do the AOT compilation, reflection points must be specified.
Issue 1
Scala's
scala.runtime.LazyValsuses reflection so it must be declared. This could be done in the Scala stdlib so that others don't need to do it themselves. But this approach is a bit brittle. Any chance of removing the use ofsun.misc.Unsafethere? If not, then I can send a PR to include the GraalVM reflection config:Issue 2
Code generated by the compiler for lazy vals also uses reflection and must be declared, like:
Also, ideally reflection is not required. But I haven't looked into what the reflection is doing. If someone uses a Scala library that uses lazy vals, and does NOT provide their reflection config for GraalVM, then the user must specify the config for the library on their own. Ideally the library authors nor the users need to specify this info.
Steps to Reproduce
https://github.com/jamesward/scala3-graalvm-tester.gitlazy-reflect-brokebranch./sbt nativeImage nativeImageRunYou will see a runtime error showing Issue 1:
(Note: It should be possible to fail the build in this case instead of producing a runtime error but there seems to be a bug: oracle/graal#4010)
If you switch to the
lazy-reflect-worksbranch and re-run./sbt nativeImage nativeImageRunyou will see that it works given the requisite config for the app in that repo that has one lazy val.One Approach (If Reflection Must Stay)
The Spring team has developed a set of annotations which provide the reflection info, like:
https://docs.spring.io/spring-native/docs/0.9.1-SNAPSHOT/api/org/springframework/nativex/hint/TypeHint.html
There has been talk that this work will merge in some way into GraalVM itself. In that case, the Scala compiler could generate the necessary
TypeHintannotations for places it generates code that uses reflection.The text was updated successfully, but these errors were encountered: