Issue on direct memory allocation in scala+sbt environment

[tglman]

OrientDB Version: 2.2.25 2.2.26-SNAPSHOT

Java Version: java 8

OS: Debian

Expected behavior

Test case: https://github.com/fixundfertig123/7547/blob/InMemoryErrorFlatSpec/src/test/scala/Test.scala goes smooth

Actual behavior

Out Of Memory on Direct Memory allocation

Steps to reproduce

Checkout the project https://github.com/fixundfertig123/7547 on the branch https://github.com/fixundfertig123/7547/tree/InMemoryErrorFlatSpec and run sbt clean + sbt compile + sbt test

from previous debug session the issue seems to come from a allocation size calculation issue:

maxPagesPerSingleArea(16384)*pageSize(65536) or preAllocationLimit(4294967296) - [bufferIndex(0)* maxPagesPerSingleArea(16384)*pageSize(65536)
Allocating of size 1048576Kb

[fixundfertig123]

@tglman Thanks! I was not very precise describing my error. Sorry for that. This is the error I get:

Aug 10, 2017 3:15:07 PM com.orientechnologies.common.log.OLogManager log
WARNUNG: Cannot find default script language for javascript
Aug 10, 2017 3:15:08 PM com.orientechnologies.common.log.OLogManager log
INFORMATION: OrientDB auto-config DISKCACHE=1.820MB (heap=1.820MB direct=1.820MB os=12.203MB), assuming maximum direct memory size equals to maximum JVM heap size
Aug 10, 2017 3:15:08 PM com.orientechnologies.common.log.OLogManager log
WARNUNG: MaxDirectMemorySize JVM option is not set or has invalid value, that may cause out of memory errors. Please set the -XX:MaxDirectMemorySize=12203m option when you start the JVM.
Start!
Start!
[info] Test:
[info] Test *** ABORTED ***
[info]   com.orientechnologies.orient.core.exception.OSchemaNotCreatedException: Schema is not created and cannot be loaded
[info]  DB name="test"
[info]   at com.orientechnologies.orient.core.metadata.schema.OSchemaShared.load(OSchemaShared.java:853)
[info]   at com.orientechnologies.orient.core.metadata.OMetadataDefault$1.call(OMetadataDefault.java:172)
[info]   at com.orientechnologies.orient.core.metadata.OMetadataDefault$1.call(OMetadataDefault.java:167)
[info]   at com.orientechnologies.common.concur.resource.OSharedContainerImpl.getResource(OSharedContainerImpl.java:62)
[info]   at com.orientechnologies.orient.core.storage.OStorageAbstract.getResource(OStorageAbstract.java:143)
[info]   at com.orientechnologies.orient.core.metadata.OMetadataDefault.init(OMetadataDefault.java:167)
[info]   at com.orientechnologies.orient.core.metadata.OMetadataDefault.load(OMetadataDefault.java:89)
[info]   at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.initAtFirstOpen(ODatabaseDocumentTx.java:3179)
[info]   at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:270)
[info]   at com.orientechnologies.orient.core.db.OPartitionedDatabasePool$DatabaseDocumentTxPooled.internalOpen(OPartitionedDatabasePool.java:448)
[info]   ...
{1=b, 2=a, 3=c, testID=1, uniqueID=1} / {1=b, 2=a, 3=c, testID=2, uniqueID=3}
Aug 10, 2017 3:15:09 PM com.orientechnologies.common.log.OLogManager log
SCHWERWIEGEND: $ANSI{green {db=test}} Error during context close for db null
com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #7:0 because the version is not the latest. Probably you are updating an old record or it has bee
n modified by another user (db=v2 your=v3)
        DB name="test"
        at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56)
        at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4315)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:3795)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4377)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1692)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
        at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2891)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2860)
        at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.shutdown(OrientBaseGraph.java:1153)
        at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.shutdown(OrientBaseGraph.java:1139)
        at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.shutdown(OrientBaseGraph.java:1132)
        at Test2$$anonfun$3.apply(Test.scala:87)
        at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
        at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
        at org.scalatest.Transformer.apply(Transformer.scala:22)
        at org.scalatest.Transformer.apply(Transformer.scala:20)
        at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1682)
        at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
        at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1685)
        at org.scalatest.FlatSpecLike$class.invokeWithFixture$1(FlatSpecLike.scala:1679)
        at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692)
        at org.scalatest.FlatSpecLike$$anonfun$runTest$1.apply(FlatSpecLike.scala:1692)
        at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
        at org.scalatest.FlatSpecLike$class.runTest(FlatSpecLike.scala:1692)
        at org.scalatest.FlatSpec.runTest(FlatSpec.scala:1685)
        at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750)
        at org.scalatest.FlatSpecLike$$anonfun$runTests$1.apply(FlatSpecLike.scala:1750)
        at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
        at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
        at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:373)
        at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:410)
        at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
        at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
        at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
        at org.scalatest.FlatSpecLike$class.runTests(FlatSpecLike.scala:1750)
        at org.scalatest.FlatSpec.runTests(FlatSpec.scala:1685)
        at org.scalatest.Suite$class.run(Suite.scala:1147)
        at org.scalatest.FlatSpec.org$scalatest$FlatSpecLike$$super$run(FlatSpec.scala:1685)
        at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795)
        at org.scalatest.FlatSpecLike$$anonfun$run$1.apply(FlatSpecLike.scala:1795)
        at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
        at org.scalatest.FlatSpecLike$class.run(FlatSpecLike.scala:1795)
        at org.scalatest.FlatSpec.run(FlatSpec.scala:1685)
        at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
        at sbt.TestRunner.runTest$1(TestFramework.scala:76)
        at sbt.TestRunner.run(TestFramework.scala:85)
        at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
        at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
        at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
        at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
        at sbt.TestFunction.apply(TestFramework.scala:207)
        at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
        at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
        at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
        at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
        at sbt.std.Transform$$anon$4.work(System.scala:63)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
        at sbt.Execute.work(Execute.scala:237)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

[info] Test2:
[info] A vertex creation
[info] - should be possible
[info] - should throw an Exception if duplicate id is entered, causing an index error on commit
[info] A different vertex creation
[info] - should be possible *** FAILED ***
[info]   com.orientechnologies.orient.core.exception.OConcurrentModificationException: Cannot UPDATE the record #7:0 because the version is not the latest. Probably you are updating an old record or it has been modified by  another user (db=v2 your=v3)
[info]  DB name="test"
[info]   at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.checkVersions(OVersionRecordConflictStrategy.java:56)
[info]   at com.orientechnologies.orient.core.conflict.OVersionRecordConflictStrategy.onUpdate(OVersionRecordConflictStrategy.java:43)
[info]   at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.checkAndIncrementVersion(OAbstractPaginatedStorage.java:4315)
[info]   at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doUpdateRecord(OAbstractPaginatedStorage.java:3795)
[info]   at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commitEntry(OAbstractPaginatedStorage.java:4377)
[info]   at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.commit(OAbstractPaginatedStorage.java:1692)
[info]   at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:541)
[info]   at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:99)
[info]   at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2891)
[info]   at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:2860)
[info]   ...
[info] Run completed in 2 seconds, 310 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 1, aborted 1
[info] Tests: succeeded 2, failed 1, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
[info] *** 1 TEST FAILED ***
[error] Failed tests:
[error]         Test2
[error] Error during tests:
[error]         Test
[error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 3 s, completed 10.08.2017 15:15:10

I am running:

• Win7 64bit
• java version "1.8.0_92" / Java(TM) SE Runtime Environment (build 1.8.0_92-b14) / Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)
• sbtVersion 0.13.15

[tglman]

hi @fixundfertig123 ,

I actually got a different error first, that we are checking, after we will check also this case.

Regards

[andrii0lomakin]

@tglman could you try again on the latest source?

[tglman]

hi @fixundfertig123,

I've been able to fix the first problem I had and reproduce the schema issue, I'm debugging to find out the cause of that problem and provide a fix.

Regards

[tglman]

Hi @fixundfertig123,

The root problem is the auto-creation of the database in concurrent way, this is not supported out of the box in 2.2.x, so probably the best way is to do a "before" operation that guarantee that the database exists before the tests run, in remote you don't have a problem because the database is required to exists in that case.

Regards

[fixundfertig123]

Hi @tglman,

thank you for your effort!!! To be honest, I am not quite sure whether I get your point. Assuming that you are referencing to the "BeforeAndAfter" traits I tried to work with that. For some reason on the new code I now get some memory exceptions I assume you fixed:

Aug 24, 2017 2:46:59 PM com.orientechnologies.common.log.OLogManager log
SCHWERWIEGEND: $ANSI{green {db=test}} Exception `75F5ED13` in storage `test`
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Unknown Source)
        at java.nio.DirectByteBuffer.<init>(Unknown Source)
        at java.nio.ByteBuffer.allocateDirect(Unknown Source)
        at com.orientechnologies.common.directmemory.OByteBufferPool.allocateBuffer(OByteBufferPool.java:328)
        at com.orientechnologies.common.directmemory.OByteBufferPool.acquireDirect(OByteBufferPool.java:279)
        at com.orientechnologies.orient.core.storage.impl.memory.ODirectMemoryOnlyDiskCache$MemoryFile.addNewPage(ODirectMemoryOnlyDiskCache.java:494)
[       at com.orientechnologies.orient.core.storage.impl.memory.ODirectMemoryOnlyDiskCache$MemoryFile.access$200(ODirectMemoryOnlyDiskCache.java:457)
info    at com.orientechnologies.orient.core.storage.impl.memory.ODirectMemoryOnlyDiskCache.allocateNewPage(ODirectMemoryOnlyDiskCache.java:246)
]       at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation.commitChanges(OAtomicOperation.java:434)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(Test2:OAtomicOperationsManager.java:468)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:412)

        at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.endAtomicOperation(ODurableComponent.java:116)
[       at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.create(OPaginatedCluster.java:195)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.addClusterInternal(OAbstractPaginatedStorage.infojava:4136)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doAddCluster(OAbstractPaginatedStorage.java:4] 117)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.create(OAbstractPaginatedStorage.java:459)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.create(ODatabaseDocumentTx.java:438)
Test2 *** ABORTED ***   at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.create(ODatabaseDocumentTx.java:398)
        at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.openDatabase(OPartitionedDatabasePool.java:305)
        at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(OPartitionedDatabasePool.java:269)

        at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.<init>(OrientBaseGraph.java:144)
        at com.tinkerpop.blueprints.impls.orient.OrientGraphNoTx.<init>(OrientGraphNoTx.java:62)
[       at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory$2.getGraph(OrientGraphFactory.java:117)
        at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getNoTx(OrientGraphFactory.java:241)
info    at Test2.<init>(Test.scala:25)
        at Test.<init>(Test.scala:14)
]       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
  com.orientechnologies.orient.core.exception.ODatabaseException: Database created with orientdb version not supported anymore,         at java.lang.reflect.Constructor.newInstance(Unknown Source
        at java.lang.Class.newInstance(Unknown Source)
        at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:435)
        at sbt.TestRunner.runTest$1(TestFramework.scala:76)

        at sbt.TestRunner.run(TestFramework.scala:85)
        at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
[       at sbt.TestFramework$$anon$2$$anonfun$$init$$1$$anonfun$apply$8.apply(TestFramework.scala:202)
        at sbt.TestFramework$.sbt$TestFramework$$withContextLoader(TestFramework.scala:185)
info    at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
        at sbt.TestFramework$$anon$2$$anonfun$$init$$1.apply(TestFramework.scala:202)
]       at sbt.TestFunction.apply(TestFramework.scala:207)
        at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
        at sbt.Tests$$anonfun$9.apply(Tests.scala:216)
        DB name="test"  at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
        at sbt.std.Transform$$anon$3$$anonfun$apply$2.apply(System.scala:44)
        at sbt.std.Transform$$anon$4.work(System.scala:63)

        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228)
[       at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
        at sbt.Execute.work(Execute.scala:237)
info    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
        at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228)
]       at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
        at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
        at java.util.concurrent.FutureTask.run(Unknown Source)
  at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.initAtFirstOpen(ODatabaseDocumentTx.java:3163)   at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

So we have some points open:

• When is your memory bug fix available so I can test?
• Is there a plan to allow this kind of "concurrent" testsuite cases? Is my kind of test so unusual?
• Since my concurrency issue is different from this memory issue labelled 7641, should I open another issue for the concurrency and that one would be something like feature request?
• I know I ask for a lot, but could you give me a practical example of how you would set up the test suite?

Best regards

[tglman]

hi @fixundfertig123,

The fix is already landed in the 2.2.x branch so it will be released with 2.2.27 in a couple of weeks, if you want to do some early try you can get the 2.2.17-SNAPSHOT from this repository: https://oss.sonatype.org/content/repositories/snapshots/

I'm not an expert on how to write MT tests in scala+sbt, but is enough to have a:

  val graphFactory: OrientGraphFactory = new OrientGraphFactory("memory:test")
  graphFactory.getTx.close()

before your current code start in MT.

Regards

[fixundfertig123]

Hi @tglman,
sorry for the late reply. and sorry, your solution did not work out. To be honest I am not an expert on SBT and test suites neither, but I got the code working. Even with version 2.2.26.

To make other save some time, I uploaded the running code at my repo: https://github.com/fixundfertig123/7547/releases/tag/WorkingTestSuite

I think crucial was changing the test suite into an object rather than a class. This hopefully helps some people and you if you recieve other SBT / SCALA / Testsuite questions.

Thanks for you effort!

[tglman]

hi @fixundfertig123,

Good to know you solved, can we close this ?

Regards

[fixundfertig123]

@tglman yes, and thanks