Flaky-test: ClassCastException in ExtensibleLoadManagerImpl.checkOwnershipAsync

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Example failures

• 2023-11-26T06:33:53.3405489Z

Exception stacktrace

org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:

 --- An unexpected error occurred in the server ---

Message: class java.util.ImmutableCollections$List12 cannot be cast to class java.util.concurrent.CompletableFuture (java.util.ImmutableCollections$List12 and java.util.concurrent.CompletableFuture are in module java.base of loader 'bootstrap')

Stacktrace:

java.lang.ClassCastException: class java.util.ImmutableCollections$List12 cannot be cast to class java.util.concurrent.CompletableFuture (java.util.ImmutableCollections$List12 and java.util.concurrent.CompletableFuture are in module java.base of loader 'bootstrap')
	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl.checkOwnershipAsync(ExtensibleLoadManagerImpl.java:625)
	at org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerWrapper.checkOwnershipAsync(ExtensibleLoadManagerWrapper.java:72)
	at org.apache.pulsar.broker.namespace.NamespaceService.lambda$checkTopicOwnership$47(NamespaceService.java:1230)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
	at org.apache.pulsar.broker.namespace.NamespaceService.checkTopicOwnership(NamespaceService.java:1230)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor$Dispatcher$ByteBuddy$sc8ugsJW.invokeWithArguments(Unknown Source)
	at org.mockito.internal.util.reflection.InstrumentationMemberAccessor.invoke(InstrumentationMemberAccessor.java:251)
	at org.mockito.internal.util.reflection.ModuleMemberAccessor.invoke(ModuleMemberAccessor.java:55)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.tryInvoke(MockMethodAdvice.java:314)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice$RealMethodCall.invoke(MockMethodAdvice.java:234)
	at org.mockito.internal.invocation.InterceptedInvocation.callRealMethod(InterceptedInvocation.java:142)
	at org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:45)
	at org.mockito.Answers.answer(Answers.java:90)
	at org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:111)
	at org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	at org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	at org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	at org.mockito.internal.creation.bytebuddy.MockMethodAdvice.handle(MockMethodAdvice.java:134)
	at org.apache.pulsar.broker.namespace.NamespaceService.checkTopicOwnership(NamespaceService.java:1228)
	at org.apache.pulsar.broker.service.BrokerService.checkTopicNsOwnership(BrokerService.java:2221)
	at org.apache.pulsar.broker.service.BrokerService.loadOrCreatePersistentTopic(BrokerService.java:1562)
	at org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$30(BrokerService.java:1086)
	at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:438)
	at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:257)
	at org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$32(BrokerService.java:1067)

[lhotari]

@heesung-sn @Demogorgon314 would you mind checking this exception?

[lhotari]

Location of the ClassCastException could searched in the detailed logs.
https://gist.githubusercontent.com/lhotari/a79c801e1e3ed88ca227db7cd93c8f8a/raw/e19c3d215f2e0b900371f3bb06a2d574bda22e14/org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImplTest.txt
part of https://gist.github.com/lhotari/a79c801e1e3ed88ca227db7cd93c8f8a which contains the surefire logs for the class

[lhotari]

This might be an issue with Mockito. Mockito isn't threadsafe in all possible ways. It's documented here how Mockito should be used: https://github.com/mockito/mockito/wiki/FAQ#is-mockito-thread-safe . Previously we have had issues #13620 and #13570 .

[lhotari]

ExtensibleLoadManagerImplTest also uses org.mockito.MockedStatic . The javadoc of that says "The mocking only affects the thread on which this static mock was created and it is not safe to use this object from another thread.".
It would be better to get rid of mocking of static methods since it won't be thread safe

[heesung-sn]

Hi, I will check this.

[heesung-sn]

Raised a PR to remove the static mock. #21668