oracle driver and serialization conflict #20396

vsevel · 2021-09-26T10:56:55Z

Describe the bug

Starting in 2.2.2, adding some serialization code and the oracle driver in the same application will lead to a java.lang.IllegalStateException: Object serialization is currently not supported when attempting to serialize objects.
This is working fine in 2.2.1.

Expected behavior

no regression. serialization code works as expected.

Actual behavior

2021-09-26 12:00:22,822 ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-0) HTTP Request to /hello/serstring failed, error id: 9464a18a-f66a-4bdf-bf3a-79d81db36dea-1: org.jboss.resteasy.spi.UnhandledException: java.lang.IllegalStateException: Object serialization is currently not supported
        at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:106)
        at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:372)
        at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:218)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:519)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$invoke$4(SynchronousDispatcher.java:261)
        at org.jboss.resteasy.core.SynchronousDispatcher.lambda$preprocess$0(SynchronousDispatcher.java:161)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
        at org.jboss.resteasy.core.SynchronousDispatcher.preprocess(SynchronousDispatcher.java:164)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:247)
        at io.quarkus.resteasy.runtime.standalone.RequestDispatcher.service(RequestDispatcher.java:73)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler.dispatch(VertxRequestHandler.java:138)
        at io.quarkus.resteasy.runtime.standalone.VertxRequestHandler$1.run(VertxRequestHandler.java:93)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$13.runWith(VertxCoreRecorder.java:543)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:29)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:29)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:829)
        at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:567)
        at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: java.lang.IllegalStateException: Object serialization is currently not supported
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:66)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:381)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1135)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
        at org.acme.getting.started.GreetingResource.serstring(GreetingResource.java:34)
        at java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:170)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:130)
        at org.jboss.resteasy.core.ResourceMethodInvoker.internalInvokeOnTarget(ResourceMethodInvoker.java:660)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTargetAfterFilter(ResourceMethodInvoker.java:524)
        at org.jboss.resteasy.core.ResourceMethodInvoker.lambda$invokeOnTarget$2(ResourceMethodInvoker.java:474)
        at org.jboss.resteasy.core.interception.jaxrs.PreMatchContainerRequestContext.filter(PreMatchContainerRequestContext.java:364)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:476)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:434)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:408)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:69)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:492)
        ... 17 more

How to Reproduce?

create a sample application.

add the following dependency:

		<dependency>
			<groupId>io.quarkus</groupId>
			<artifactId>quarkus-jdbc-oracle</artifactId>
		</dependency>

add the following build args:

<quarkus.native.additional-build-args>--initialize-at-run-time=oracle.jdbc.datasource.impl.OracleDataSource,--allow-incomplete-classpath</quarkus.native.additional-build-args>

add the following service:

    @GET
    @Produces(MediaType.TEXT_PLAIN)
    @Path("/serstring")
    public String serstring() throws IOException, ClassNotFoundException {

        byte[] bytes = null;

        try (
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ObjectOutputStream oos = new ObjectOutputStream(baos)) {

            oos.writeObject("Hello RESTEasy");
            oos.flush();
            bytes = baos.toByteArray();
        }

        try (ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
                ObjectInputStream ois = new ObjectInputStream(bais)) {
            return (String) ois.readObject();
        }
    }

and the following test:

    @Test
    public void testser() {
        given()
                .when().get("/hello/serstring")
                .then()
                .statusCode(200)
                .body(is("Hello RESTEasy"));
    }

launch a native build, in 2.2.1 it works, in 2.2.2 it fails. it fails also on 2.3.0.CR1.

Output of `uname -a` or `ver`

No response

Output of `java -version`

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2.2.2

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

No response

Additional information

No response

The text was updated successfully, but these errors were encountered:

vsevel · 2021-09-26T10:58:01Z

most likely introduced by #19928
/cc @zakkak @Sanne

Sanne · 2021-09-26T20:19:08Z

Hi @vsevel , thanks for the report!

Could you clarify why you're setting <quarkus.native.additional-build-args>--initialize-at-run-time=oracle.jdbc.datasource.impl.OracleDataSource,--allow-incomplete-classpath</quarkus.native.additional-build-args> ?

vsevel · 2021-09-27T09:35:44Z

hi @Sanne, if I don't, I get an error related tocom.sun.jmx.mbeanserver.JmxMBeanServer.

when I specify:

<quarkus.native.additional-build-args>initialize-at-run-time=com.sun.jmx.mbeanserver.JmxMBeanServer,--trace-class-initialization=com.sun.jmx.mbeanserver.JmxMBeanServer</quarkus.native.additional-build-args>

I get this build error:

Error: Classes that should be initialized at run time got initialized during image building:
 com.sun.jmx.mbeanserver.JmxMBeanServer the class was requested to be initialized at run time (from the command line with 'com.sun.jmx.mbeanserver.JmxMBeanServer'). oracle.jdbc.datasource.impl.OracleDataSource caused initialization of this class with the following trace:
        at com.sun.jmx.mbeanserver.JmxMBeanServer.<clinit>(JmxMBeanServer.java)
        at javax.management.MBeanServerBuilder.newMBeanServerDelegate(MBeanServerBuilder.java:66)
        at javax.management.MBeanServerFactory.newMBeanServer(MBeanServerFactory.java:321)
        at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:231)
        at javax.management.MBeanServerFactory.createMBeanServer(MBeanServerFactory.java:192)
        at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:483)
        at oracle.jdbc.datasource.impl.OracleDataSource.registerMBean(OracleDataSource.java:2394)
        at oracle.jdbc.datasource.impl.OracleDataSource$3.run(OracleDataSource.java:2361)
        at java.security.AccessController.doPrivileged(Unknown Source)
        at oracle.jdbc.datasource.impl.OracleDataSource.<clinit>(OracleDataSource.java:2359)

vsevel · 2021-09-27T10:09:32Z

I am not sure about the --allow-incomplete-classpath. I do not know why it was added, and why. my different tests seem to work fine without it. may be it is not needed anymore.

vsevel · 2021-09-27T12:43:26Z

@Sanne I was able to drop --allow-incomplete-classpath with a little work on my side. so unrelated to the present issue.

vsevel · 2021-10-26T16:49:26Z

I was able to reproduce in a small example, without Quarkus, just using native-image.

Here is the main program:

public class Main {
    public static void main(String[] args) throws IOException, ClassNotFoundException {
        ser();
        // conn();
        System.out.println("Done");
    }

    static void ser() throws IOException, ClassNotFoundException {

        System.out.println("Serialization");

        byte[] bytes = null;
        try (
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ObjectOutputStream oos = new ObjectOutputStream(baos)) {
            oos.writeObject("hello");
            oos.flush();
            bytes = baos.toByteArray();
        }

        try (
                ByteArrayInputStream bais = new ByteArrayInputStream(bytes);
                ObjectInputStream ois = new ObjectInputStream(bais)) {
            String s = (String) ois.readObject();
            System.out.println("ser => " + s);
        }
    }

    //    static void conn() throws ClassNotFoundException {
    //        System.out.println("Connection");
    //        Class.forName(OracleDriver.class.getName());
    //    }
}

And pom:

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		 xmlns="http://maven.apache.org/POM/4.0.0"
		 xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>test</groupId>
	<artifactId>native-image.example</artifactId>
	<version>1.0-SNAPSHOT</version>
	<properties>
		<maven.compiler.source>11</maven.compiler.source>
		<maven.compiler.target>11</maven.compiler.target>
	</properties>
	<dependencies>
		<dependency>
			<groupId>com.oracle.database.jdbc</groupId>
			<artifactId>ojdbc11</artifactId>
			<version>21.3.0.0</version>
		</dependency>
	</dependencies>
</project>

To reproduce:

mvn clean package
# export CP=target/native-image.example-1.0-SNAPSHOT.jar
export CP=target/native-image.example-1.0-SNAPSHOT.jar:../../maven-local-repository/com/oracle/database/jdbc/ojdbc11/21.3.0.0/ojdbc11-21.3.0.0.jar
java -agentlib:native-image-agent=config-output-dir=target/config/META-INF/native-image -cp $CP test.Main
$GRAALVM_HOME/bin/native-image -J-Djava.io.tmpdir=/home/sevel/tmpdir -cp $CP:target/config test.Main
./test.main

The execution gives:

[sevel@graalvmdev native-image]$ ./test.main
Serialization
Exception in thread "main" java.lang.IllegalStateException: Object serialization is currently not supported
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:66)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:381)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1135)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
        at test.Main.ser(Main.java:26)
        at test.Main.main(Main.java:14)

if I change the classpath to export CP=target/native-image.example-1.0-SNAPSHOT.jar, then everything works fine:

[sevel@graalvmdev native-image]$ ./test.main
Serialization
ser => hello
Done

So the issue arises by just having the ojdbc11 in the classpath. No need to actually use the classes.

I can see the driver has a bunch of config files in META-INF/native-image and some auto features in oracle.nativeimage, which are probably getting in the way of serialization (although I am not seeing anything suspicious at first sight).

I am using:

[sevel@graalvmdev native-image]$ $GRAALVM_HOME/bin/native-image --version
GraalVM 21.2.0 Java 11 CE (Java Version 11.0.12+6-jvmci-21.2-b08)

and oracle driver ojdbc11:21.3.0.0

@Sanne I know it is not a Quarkus issue, but how do you think we could make progress on this? thanks.

vsevel · 2021-10-26T16:52:44Z

I missed it initially, but it is simply caused by:

@TargetClass(
    className = "java.io.ObjectStreamClass"
)
final class Target_java_io_ObjectStreamClass {
    @Substitute
    private Target_java_io_ObjectStreamClass() {
        throw new IllegalStateException("Object serialization is currently not supported");
    }

    @Substitute
    private Target_java_io_ObjectStreamClass(Class<?> var1) {
        throw new IllegalStateException("Object serialization is currently not supported");
    }
}

Is there any way we can work around this in Quarkus?

vsevel · 2021-10-26T17:14:25Z

I extracted the content of the driver jar, removed oracle\nativeimage\Target_java_io_ObjectStreamClass.class, and repackaged the jar. With that new jar I was able to get my simple test to run:

[sevel@graalvmdev native-image]$ ./test.main
Serialization
ser => hello
Connection
conn=oracle.jdbc.driver.T4CConnection@2f52cc7
Done

Using this additional code in the program:

    static void conn() throws Exception {
        System.out.println("Connection");
        Class.forName(OracleDriver.class.getName());

        Connection conn = DriverManager.getConnection("jdbc:oracle:thin:@localhost:49161:xe", "system", "oracle");
        System.out.println("conn=" + conn);
        conn.close();
    }

It does not mean the hack is all what is needed.

Is there any way you could get oracle to produce a new driver? thanks.

Sanne · 2021-10-27T10:07:48Z

hi @vsevel - thank you very much, such a simple reproducer is great to have.

I'll try to look at it today but I'm not sure I can do much about it... just curious to learn more about it.

Regarding the --allow-incomplete-classpath flags and its use in the Oracle driver: we've discussed this again with the GraalVM team and the native-image team is giving this matter attention; it's technically much more complex than what I expected though, deeply related with class initialization strategies so while we made great progress with brainstorming options, I'm afraid it will take some more time.

See also oracle/graal#3932 , oracle/graal#2762 , oracle/graal#3929

Added some more notes just now:

Native Image Committer Community Meeting 2021-10-20 oracle/graal#3932 (comment)

vsevel · 2021-10-27T16:08:17Z

thanks for the update @Sanne

just to clarify:

I do not have a particular point about --allow-incomplete-classpath at this time with regard to persistence as noted in oracle driver and serialization conflict #20396 (comment)
However, I had to add a --initialize-at-run-time=oracle.jdbc.datasource.impl.OracleDataSource as noted in oracle driver and serialization conflict #20396 (comment) - not a blocking issue
And I have a blocking issue with regard to the Target_java_io_ObjectStreamClass substitution in oracle driver version 21.3.0.0

to investigate the issue further, I repackaged a new oracle driver without the Target_java_io_ObjectStreamClass substitution class, and used in my business application, and I can confirm that on the tested use cases (involving hibernate and serialization) it works successfully.

if I have not missed anything, we only need a version of the driver without Target_java_io_ObjectStreamClass, or a workaround in Quarkus to get this class to be ignored (I do not know if this possible).

Sanne · 2021-10-27T16:22:56Z

But where is this Target_java_io_ObjectStreamClass class located?

vsevel · 2021-10-27T16:39:51Z

in the oracle driver

Sanne · 2021-10-28T14:17:34Z

i.c. thanks... I didn't expect that at all, it's concerning that a library makes the unilateral decision to disable JDK shared functionality rather than patching their own code.

I suppose it was hard for them to fix and the previous GraalVM version didn't support serialization, but still it seems a mistake to include hardcoded substitutions that make an assumption about which version of GraalVM will be used - that's not for a library to decide.

I've opened #21069 as one aspect to handle such cases.

Not sure how to handle this specific problem ... honestly I think your best option is to open a support request with Oracle and ask them to fix the driver.

vsevel · 2021-10-28T16:40:57Z

Not sure how to handle this specific problem ... honestly I think your best option is to open a support request with Oracle and ask them to fix the driver.

yes. right. I was hoping you may have some weight on getting them to do things right, specifically in the context of graalvm and the Native Image Committer Community Meeting. it might a good topic to bring to @christianwimmer.
but yes, in parallel, I am going to pursue the support case option through our corporate contract.
we will see who is the most effective ;)

vsevel · 2021-12-03T10:31:04Z

fyi, the issue has been reproduced by oracle support. a bug should be submitted soon to the dev team.

Sanne · 2021-12-03T23:48:38Z

you motivated me to have another look, and it turned out it wasn't that complicated to workaround :)

See #21929

vsevel · 2021-12-04T09:45:20Z

excellent @Sanne
that is exactly how I got it to work: unpacked the jars, removed the class, and repackaged. I can confirm that at least in my business app with limited testing I was successful with no side effects.
that is one thing I was wondering about. can we substitute a substitution? in that case, it was straightforward because we could remove the entire resource. but I was thinking if this was juste one method we wanted to revert, how would we do this?
anyway this is a clean solution. good job!
I will continue to track the bug on the oracle side. and when it is fixed and released, I will get that information back into quarkus code, so that eventually we can remove your workaround.

Sanne · 2021-12-04T21:46:52Z

excellent @Sanne that is exactly how I got it to work: unpacked the jars, removed the class, and repackaged. I can confirm that at least in my business app with limited testing I was successful with no side effects.

Right, this patch will do the same.

I should add that I had concerns about following this approach of blindly modifying a non-OSS library as we couldn't know all substitutions and their nature, but since I've been working on a POC for #21069 and while doing it, it became clearer that the tooling can generate enough insight for this to be controllable.

And thanks for your test!

that is one thing I was wondering about. can we substitute a substitution? in that case, it was straightforward because we could remove the entire resource. but I was thinking if this was juste one method we wanted to revert, how would we do this?

Right this was simpe today, it could get uglier. But we can always delete a substitution and provide an alternative in our extenstions; or we can also actually modify bytecode with a different build item. It's similar to "substitutions" but it works on the bytecode level so we can even modify things in such a way to have effect in JVM mode - potentially.

Hopefully we'll never need to do such things again, but it's reassuring to see that our transformations are this powerful and flexible so that in case of need there's always a way out. We'll of course always favour upstream patching, but it's good to have ways to expedite things.

anyway this is a clean solution. good job! I will continue to track the bug on the oracle side. and when it is fixed and released, I will get that information back into quarkus code, so that eventually we can remove your workaround.

Thanks! We'll also keep working on #21069 to help spotting new unexpected subtitutions, and possibly watch for such ones which we expet to get removed. And we definitely need to keep working with the teams maintaining the upstreams, such as Oracle JDBC in thi case.

vsevel · 2022-01-31T14:40:18Z

FYI, oracle support says: The development team has fixed the issue and they are working on the next steps to release the fix.

Sanne · 2022-01-31T16:32:10Z

Awesome, many thanks @vsevel for making that happen!

vsevel · 2022-03-21T15:07:13Z

the fix will be in version 21.6. the driver should be available mid-may.
I see we are still in 21.4: https://github.com/quarkusio/quarkus/blob/main/bom/application/pom.xml#L128
fyi, 21.5 is already available:

Release Date Version Download Link Included in Windows Bundle
18-Jan-2022 Database Release Update 21.5.0 Patch [33516412]()

Sanne · 2022-03-22T10:36:13Z

thanks @vsevel , tracking that as #24468

vsevel added the kind/bug Something isn't working label Sep 26, 2021

quarkus-bot bot added the triage/needs-triage label Sep 26, 2021

Sanne removed the triage/needs-triage label Oct 28, 2021

JiriOndrusek mentioned this issue Nov 5, 2021

Sql tests with external oracle database fail apache/camel-quarkus#3256

Closed

Sanne mentioned this issue Dec 3, 2021

Oracle JDBC driver presence to not break JVM's Serialization support in native image #21929

Merged

geoand closed this as completed in #21929 Dec 4, 2021

quarkus-bot bot added this to the 2.6 - main milestone Dec 4, 2021

gsmet modified the milestones: 2.6.0.CR1, 2.5.2.Final Dec 9, 2021

aldettinger mentioned this issue Dec 14, 2021

Sql doc update because of the failure with external oracle database #… apache/camel-quarkus#3272

Merged

Sanne mentioned this issue Mar 22, 2022

Upgrade Oracle JDBC to 21.5.0.0 #24468

Closed

a483210 mentioned this issue Mar 30, 2023

oracle driver and serialization conflict spring-projects/spring-boot#34819

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oracle driver and serialization conflict #20396

oracle driver and serialization conflict #20396

vsevel commented Sep 26, 2021

vsevel commented Sep 26, 2021

Sanne commented Sep 26, 2021

vsevel commented Sep 27, 2021

vsevel commented Sep 27, 2021

vsevel commented Sep 27, 2021

vsevel commented Oct 26, 2021

vsevel commented Oct 26, 2021

vsevel commented Oct 26, 2021

Sanne commented Oct 27, 2021

vsevel commented Oct 27, 2021

Sanne commented Oct 27, 2021

vsevel commented Oct 27, 2021

Sanne commented Oct 28, 2021 •

edited

Loading

vsevel commented Oct 28, 2021

vsevel commented Dec 3, 2021

Sanne commented Dec 3, 2021

vsevel commented Dec 4, 2021

Sanne commented Dec 4, 2021

vsevel commented Jan 31, 2022

Sanne commented Jan 31, 2022

vsevel commented Mar 21, 2022

Sanne commented Mar 22, 2022

oracle driver and serialization conflict #20396

oracle driver and serialization conflict #20396

Comments

vsevel commented Sep 26, 2021

Describe the bug

Expected behavior

Actual behavior

How to Reproduce?

Output of uname -a or ver

Output of java -version

GraalVM version (if different from Java)

Quarkus version or git rev

Build tool (ie. output of mvnw --version or gradlew --version)

Additional information

vsevel commented Sep 26, 2021

Sanne commented Sep 26, 2021

vsevel commented Sep 27, 2021

vsevel commented Sep 27, 2021

vsevel commented Sep 27, 2021

vsevel commented Oct 26, 2021

vsevel commented Oct 26, 2021

vsevel commented Oct 26, 2021

Sanne commented Oct 27, 2021

vsevel commented Oct 27, 2021

Sanne commented Oct 27, 2021

vsevel commented Oct 27, 2021

Sanne commented Oct 28, 2021 • edited Loading

vsevel commented Oct 28, 2021

vsevel commented Dec 3, 2021

Sanne commented Dec 3, 2021

vsevel commented Dec 4, 2021

Sanne commented Dec 4, 2021

vsevel commented Jan 31, 2022

Sanne commented Jan 31, 2022

vsevel commented Mar 21, 2022

Sanne commented Mar 22, 2022

Output of `uname -a` or `ver`

Output of `java -version`

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Sanne commented Oct 28, 2021 •

edited

Loading