DeadLock when saving an Object #7172

sbespalov · 2017-02-15T03:21:17Z

OrientDB Version: 2.2.16

Java Version: 1.8.0_65

OS: Ubuntu 16.04

Please help to resolve it.

Stack:

Thread [main] (Suspended)	
	owns: NoProxyRepositoryImpl<T>  (id=63)	
	Unsafe.park(boolean, long) line: not available [native method]	
	LockSupport.park(Object) line: 175	
	ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).parkAndCheckInterrupt() line: 836	
	ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquireQueued(AbstractQueuedSynchronizer$Node, int) line: 870	
	ReentrantLock$NonfairSync(AbstractQueuedSynchronizer).acquire(int) line: 1199	
	ReentrantLock$NonfairSync.lock() line: 209	
	ReentrantLock.lock() line: 285	
	OAdaptiveLock.lock() line: 100	
	OChannelBinaryAsynchClient(OChannel).acquireReadLock() line: 129	
	OChannelBinaryAsynchClient.beginResponse(int, long, boolean) line: 178	
	OChannelBinaryAsynchClient.beginResponse(int, boolean) line: 167	
	OStorageRemote.beginResponse(OChannelBinaryAsynchClient, OStorageRemoteSession) line: 2205	
	OStorageRemote$6.execute(OChannelBinaryAsynchClient, OStorageRemoteSession) line: 553	
	OStorageRemote$6.execute(OChannelBinaryAsynchClient, OStorageRemoteSession) line: 547	
	OStorageRemote$1.execute(OChannelBinaryAsynchClient, OStorageRemoteSession) line: 165	
	OStorageRemote.baseNetworkOperation(OStorageRemoteOperation<T>, String, int) line: 238	
	OStorageRemote.asyncNetworkOperation(OStorageRemoteOperationWrite, OStorageRemoteOperationRead<T>, int, ORecordId, ORecordCallback<T>, String) line: 157	
	OStorageRemote.createRecord(ORecordId, byte[], int, byte, int, ORecordCallback<Long>) line: 533	
	OPartitionedDatabasePool$DatabaseDocumentTxPooled(ODatabaseDocumentTx).executeSaveRecord(ORecord, String, int, OPERATION_MODE, boolean, ORecordCallback<Number>, ORecordCallback<Integer>) line: 2184	
	OTransactionNoTx.saveNew(ODocument, ODirtyManager, String, ORecord, OPERATION_MODE, boolean, ORecordCallback<Number>, ORecordCallback<Integer>) line: 235	
	OTransactionNoTx.saveRecord(ORecord, String, OPERATION_MODE, boolean, ORecordCallback<Number>, ORecordCallback<Integer>) line: 168	
	OPartitionedDatabasePool$DatabaseDocumentTxPooled(ODatabaseDocumentTx).save(ORecord, String, OPERATION_MODE, boolean, ORecordCallback<Number>, ORecordCallback<Integer>) line: 2716	
	OObjectDatabaseTx.save(Object, String, OPERATION_MODE, boolean, ORecordCallback<Number>, ORecordCallback<Integer>) line: 464	
	OObjectDatabaseTx.save(Object) line: 398	
	OrientObjectTemplate(AbstractOrientOperations<T>).save(S) line: 331	
	NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]	
	NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62	
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
	Method.invoke(Object, Object...) line: 497	
	AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 333	
	JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 207	
	$Proxy40.save(Object) line: not available	
	SimpleOrientStrategy<T>.save(S) line: 20	
	NoProxyRepositoryImpl<T>(SimpleOrientRepository<T>).save(S) line: 91	
	NoProxyRepositoryImpl<T>.save(S) line: 46	
	NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]	
	NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62	
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
	Method.invoke(Object, Object...) line: 497	
	RepositoryFactorySupport$QueryExecutorMethodInterceptor.executeMethodOn(Object, Method, Object[]) line: 483	
	RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(MethodInvocation) line: 468	
	RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(MethodInvocation) line: 440	
	ReflectiveMethodInvocation.proceed() line: 179	
	DefaultMethodInvokingMethodInterceptor.invoke(MethodInvocation) line: 61	
	ReflectiveMethodInvocation.proceed() line: 179	
	TransactionInterceptor$1.proceedWithInvocation() line: 99	
	TransactionInterceptor(TransactionAspectSupport).invokeWithinTransaction(Method, Class<?>, InvocationCallback) line: 282	
	TransactionInterceptor.invoke(MethodInvocation) line: 96	
	ReflectiveMethodInvocation.proceed() line: 179	
	PersistenceExceptionTranslationInterceptor.invoke(MethodInvocation) line: 136	
	ReflectiveMethodInvocation.proceed() line: 179	
	ExposeInvocationInterceptor.invoke(MethodInvocation) line: 92	
	ReflectiveMethodInvocation.proceed() line: 179	
	JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 213	
	$Proxy68.save(Object) line: not available	
	ArtifactEntryServiceImpl.save(S) line: 125	
	ArtifactEntryServiceImpl.save(Object) line: 32	
	NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]	
	NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62	
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
	Method.invoke(Object, Object...) line: 497	
	AopUtils.invokeJoinpointUsingReflection(Object, Method, Object[]) line: 333	
	ReflectiveMethodInvocation.invokeJoinpoint() line: 190	
	ReflectiveMethodInvocation.proceed() line: 157	
	TransactionInterceptor$1.proceedWithInvocation() line: 99	
	TransactionInterceptor(TransactionAspectSupport).invokeWithinTransaction(Method, Class<?>, InvocationCallback) line: 282	
	TransactionInterceptor.invoke(MethodInvocation) line: 96	
	ReflectiveMethodInvocation.proceed() line: 179	
	JdkDynamicAopProxy.invoke(Object, Method, Object[]) line: 213	
	$Proxy69.save(Object) line: not available	
	ArtifactEntryServiceTest.createArtifactEntry(ArtifactCoordinates, String, String) line: 177	
	ArtifactEntryServiceTest.createArtifacts(String, String, String, String) line: 160	
	ArtifactEntryServiceTest.searchByTwoCoordinate() line: 93	
	NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]	
	NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62	
	DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43	
	Method.invoke(Object, Object...) line: 497	
	FrameworkMethod$1.runReflectiveCall() line: 50	
	FrameworkMethod$1(ReflectiveCallable).run() line: 12	
	FrameworkMethod.invokeExplosively(Object, Object...) line: 47	
	InvokeMethod.evaluate() line: 17	
	RunBeforeTestMethodCallbacks.evaluate() line: 75	
	RunAfterTestMethodCallbacks.evaluate() line: 86	
	SpringRepeat.evaluate() line: 84	
	SpringJUnit4ClassRunner(ParentRunner<T>).runLeaf(Statement, Description, RunNotifier) line: 325	
	SpringJUnit4ClassRunner.runChild(FrameworkMethod, RunNotifier) line: 252	
	SpringJUnit4ClassRunner.runChild(Object, RunNotifier) line: 94	
	ParentRunner$3.run() line: 290	
	ParentRunner$1.schedule(Runnable) line: 71	
	SpringJUnit4ClassRunner(ParentRunner<T>).runChildren(RunNotifier) line: 288	
	ParentRunner<T>.access$000(ParentRunner, RunNotifier) line: 58	
	ParentRunner$2.evaluate() line: 268	
	RunBeforeTestClassCallbacks.evaluate() line: 61	
	RunAfterTestClassCallbacks.evaluate() line: 70	
	SpringJUnit4ClassRunner(ParentRunner<T>).run(RunNotifier) line: 363	
	SpringJUnit4ClassRunner.run(RunNotifier) line: 191	
	JUnit4TestReference.run(TestExecution) line: 86	
	TestExecution.run(ITestReference[]) line: 38	
	RemoteTestRunner.runTests(String[], String, TestExecution) line: 459	
	RemoteTestRunner.runTests(TestExecution) line: 678	
	RemoteTestRunner.run() line: 382	
	RemoteTestRunner.main(String[]) line: 192

The text was updated successfully, but these errors were encountered:

andrii0lomakin · 2017-02-15T07:47:46Z

@sbespalov could you provide full thread dump? We can not detect deadlock by a single thread.
Also, could you use gist for that?

sbespalov · 2017-02-15T08:04:31Z

@Laa, thank you for quick response
thread dump is here

andrii0lomakin · 2017-02-15T09:25:16Z

@sbespalov why do you think there is deadlock there could you provide me your reasoning?

sbespalov · 2017-02-15T09:32:03Z

@Laa, because application hangs here LockSupport.park(Object) line: 175 and seems that it waits for someone to release com.orientechnologies.orient.enterprise.channel.OChannel.lockRead.

andrii0lomakin · 2017-02-15T09:40:57Z

@sbespalov so does thread which you pointed hang there forever? I mean if your test or task is finished and a database is idle you see such hanging thread, do not you?

andrii0lomakin · 2017-02-15T09:41:29Z

@tglman could you look at this ?

sbespalov · 2017-02-15T09:56:30Z

@sbespalov so does thread which you pointed hang there forever? I mean if your test or task is finished and a database is idle you see such hanging thread, do not you?

@Laa, It's an UnitTest and it has only one thread "main" #1 prio=5 os_prio=0 tid=0x00007fcd7c012000 nid=0x27bc waiting on condition [0x00007fcd836fa000] which hangs forever, so my test can not be finished. All the DB operations performed one by one in this thread.

tglman · 2017-02-15T10:50:58Z

hi @sbespalov,

from what i see the client is just waiting for the server to answer, could you take the thread dump of the server ? or anyway what's the status of the server when you run the specific test case ?

Just as a info, the thread is waiting on a lock because there is another thread named as "Asynch Client" getting the data from the network for him.

Regards

sbespalov · 2017-02-15T12:09:22Z

@tglman, hi

Client and Server running under same JVM, so dump above must also contains Server threads too.

tglman · 2017-02-15T12:18:51Z

Hi,

I see, well so this thread:

"OrientDB (/127.0.0.1:2424) <- BinaryClient (/127.0.0.1:47454)" #17 daemon prio=5 os_prio=0 tid=0x00007fcd10001800 nid=0x27dc runnable [0x00007fcd40100000]
   java.lang.Thread.State: RUNNABLE
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.isClosed(ODatabaseDocumentTx.java:1371)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.checkOpeness(ODatabaseDocumentTx.java:3103)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2120)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:238)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:168)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2716)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2585)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:103)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.createRecord(ONetworkProtocolBinary.java:2769)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.createRecord(ONetworkProtocolBinary.java:1748)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:562)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:336)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:200)
at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

it seems to be the one that is handling the request and is not blocked, so i don't think there is a deadlock, can you send us the test case or "cleared version" of it so we can check it?

Regards

sbespalov · 2017-02-15T12:36:09Z

"cleared version" can take some time, I will try to reproduce it separately if needed
But, if you are able, you can try to checkout our project and run tests with Maven
The target project version can be found here, you can run mvn clean install to execute test cases (some of them will hang with this DeadLock)

tglman · 2017-02-15T13:37:10Z

hi @sbespalov,

yes i'll try that.

Regards

tglman · 2017-02-15T14:32:12Z

hi @sbespalov,
I tried to build it, i resolved some snapshot dependencies but i got stack on jnuget i could not find it, is it possible to get a branch that build without that snapshots, (or at least without the jnuget snapshot)?

Regards

sbespalov · 2017-02-16T01:30:17Z

Oh, I am very sorry for that.
The simplest way is to use our MavenRepo to build the project.
Just define the profile below in your settings.xml, and then use it with mvn clean install -P carlspring-repositories

<profiles>
		<profile>
			<id>carlspring-repositories</id>
			<repositories>
				<repository>
					<id>carlspring</id>
					<name>carlspring</name>
					<url>http://dev.carlspring.org/nexus/content/groups/carlspring</url>
					<layout>default</layout>
				</repository>
			</repositories>
			<pluginRepositories>
				<pluginRepository>
					<id>carlspring</id>
					<name>carlspring</name>
					<url>http://dev.carlspring.org/nexus/content/groups/carlspring</url>
					<layout>default</layout>
				</pluginRepository>
			</pluginRepositories>
		</profile>
	</profiles>

sbespalov · 2017-02-16T02:53:06Z

Just for information
I am using OPartitionedDatabasePool to get DB connections and OServer to start the server.

tglman · 2017-02-16T10:29:30Z

hi @sbespalov,

I reproduced it, is not a deadlock but something is wrong in there, checking.

Regards

sbespalov · 2017-02-16T11:24:25Z

Great!
Waiting for your response

sbespalov · 2017-02-17T04:25:25Z

hi @tglman
Is there any help, needed from my side?

tglman · 2017-02-17T09:20:16Z

hi @sbespalov,

I figure out the source of the problem, and is in the Object Database layer, I'm trying to write a test case that reproduce it in a determistic way and fix it.

Regards

sbespalov · 2017-02-20T06:03:47Z

@tglman
I do not want to rush you, but when the fix is expected to be available?

… issue #7172

tglman · 2017-02-20T12:15:09Z

hi @sbespalov,

found the root cause, done the test case for the root cause and fixed, i double checked also with your test case and it work correctly, it will be released in 2.2.18, if you want you can try it out in 2.2.18-SNAPSHOT.

Closing this.

Regards

carlspring · 2017-02-21T01:03:59Z

@tglman : Many thanks for helping us with this! :)

sbespalov · 2017-02-21T01:15:23Z

Many thanks guys!
You help a lot!

metapp · 2017-05-19T12:59:34Z

I'm experiencing the same problem as @sbespalov did, even on version 2.2.20. I am using the vertx framework along with orientdb. I get this error only when saving a second entity (different cluster/class but same database) the first save goes through fine. I'm not sure if the first save locks the db. I have included my stacktrace below. I would appreciate any help on this. I have been struggling for hours:

May 19, 2017 2:59:37 PM io.vertx.core.impl.BlockedThreadChecker
WARNING: Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 91752 ms, time limit is 60000
io.vertx.core.VertxException: Thread blocked
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at com.orientechnologies.common.concur.lock.OAdaptiveLock.lock(OAdaptiveLock.java:100)
at com.orientechnologies.orient.enterprise.channel.OChannel.acquireReadLock(OChannel.java:127)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:164)
at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:153)
at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:2133)
at com.orientechnologies.orient.client.remote.OStorageRemote$6.execute(OStorageRemote.java:543)
at com.orientechnologies.orient.client.remote.OStorageRemote$6.execute(OStorageRemote.java:537)
at com.orientechnologies.orient.client.remote.OStorageRemote$1.execute(OStorageRemote.java:166)
at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:230)
at com.orientechnologies.orient.client.remote.OStorageRemote.asyncNetworkOperation(OStorageRemote.java:158)
at com.orientechnologies.orient.client.remote.OStorageRemote.createRecord(OStorageRemote.java:523)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2075)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:246)
at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:179)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2598)
at com.orientechnologies.orient.object.db.OObjectDatabaseTx.save(OObjectDatabaseTx.java:465)
at com.orientechnologies.orient.object.db.OObjectDatabaseTx.save(OObjectDatabaseTx.java:399)

andrii0lomakin · 2017-05-19T13:16:23Z

@metapp could you send us complete stack trace as separate gist?

andrii0lomakin added the waiting reply label Feb 15, 2017

prjhub removed the waiting reply label Feb 15, 2017

andrii0lomakin assigned tglman Feb 15, 2017

andrii0lomakin added this to the 2.2.x (next hotfix) milestone Feb 15, 2017

tglman added a commit that referenced this issue Feb 20, 2017

fixed issue of simple linked serializable objects in object database,…

5c9433a

… issue #7172

tglman added a commit that referenced this issue Feb 20, 2017

fixed issue of simple linked serializable objects in object database,…

6792f77

… issue #7172

tglman closed this as completed Feb 20, 2017

santo-it modified the milestones: 2.2.18, 2.2.x (next hotfix) Feb 20, 2017

carlspring mentioned this issue Mar 17, 2017

SB-731: Hook up the code for SB-711 with the respective service class strongbox/strongbox#224

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DeadLock when saving an Object #7172

DeadLock when saving an Object #7172

sbespalov commented Feb 15, 2017 •

edited

Loading

andrii0lomakin commented Feb 15, 2017

sbespalov commented Feb 15, 2017

andrii0lomakin commented Feb 15, 2017 •

edited

Loading

sbespalov commented Feb 15, 2017

andrii0lomakin commented Feb 15, 2017

andrii0lomakin commented Feb 15, 2017

sbespalov commented Feb 15, 2017

tglman commented Feb 15, 2017

sbespalov commented Feb 15, 2017

tglman commented Feb 15, 2017

sbespalov commented Feb 15, 2017 •

edited

Loading

tglman commented Feb 15, 2017

tglman commented Feb 15, 2017

sbespalov commented Feb 16, 2017

sbespalov commented Feb 16, 2017

tglman commented Feb 16, 2017

sbespalov commented Feb 16, 2017

sbespalov commented Feb 17, 2017 •

edited

Loading

tglman commented Feb 17, 2017

sbespalov commented Feb 20, 2017 •

edited

Loading

tglman commented Feb 20, 2017

carlspring commented Feb 21, 2017

sbespalov commented Feb 21, 2017

metapp commented May 19, 2017

andrii0lomakin commented May 19, 2017

DeadLock when saving an Object #7172

DeadLock when saving an Object #7172

Comments

sbespalov commented Feb 15, 2017 • edited Loading

OrientDB Version: 2.2.16

Java Version: 1.8.0_65

OS: Ubuntu 16.04

andrii0lomakin commented Feb 15, 2017

sbespalov commented Feb 15, 2017

andrii0lomakin commented Feb 15, 2017 • edited Loading

sbespalov commented Feb 15, 2017

andrii0lomakin commented Feb 15, 2017

andrii0lomakin commented Feb 15, 2017

sbespalov commented Feb 15, 2017

tglman commented Feb 15, 2017

sbespalov commented Feb 15, 2017

tglman commented Feb 15, 2017

sbespalov commented Feb 15, 2017 • edited Loading

tglman commented Feb 15, 2017

tglman commented Feb 15, 2017

sbespalov commented Feb 16, 2017

sbespalov commented Feb 16, 2017

tglman commented Feb 16, 2017

sbespalov commented Feb 16, 2017

sbespalov commented Feb 17, 2017 • edited Loading

tglman commented Feb 17, 2017

sbespalov commented Feb 20, 2017 • edited Loading

tglman commented Feb 20, 2017

carlspring commented Feb 21, 2017

sbespalov commented Feb 21, 2017

metapp commented May 19, 2017

andrii0lomakin commented May 19, 2017

sbespalov commented Feb 15, 2017 •

edited

Loading

andrii0lomakin commented Feb 15, 2017 •

edited

Loading

sbespalov commented Feb 15, 2017 •

edited

Loading

sbespalov commented Feb 17, 2017 •

edited

Loading

sbespalov commented Feb 20, 2017 •

edited

Loading