Batch insert sql: Direct buffer memory

[zlfsimon]

OrientDB Version: 2.2.5

Java Version: 1.8.0_144

OS: centos 7 | mac

Exception

Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:694)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at com.orientechnologies.common.directmemory.OByteBufferPool.allocateBuffer(OByteBufferPool.java:328)
	at com.orientechnologies.common.directmemory.OByteBufferPool.acquireDirect(OByteBufferPool.java:279)
	at com.orientechnologies.orient.core.storage.cache.local.OWOWCache.load(OWOWCache.java:769)
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.updateCache(O2QCache.java:1149)
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.doLoad(O2QCache.java:347)
	at com.orientechnologies.orient.core.storage.cache.local.twoq.O2QCache.allocateNewPage(O2QCache.java:398)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperation.commitChanges(OAtomicOperation.java:434)
	at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:468)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doCreateRecord(OAbstractPaginatedStorage.java:3732)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.createRecord(OAbstractPaginatedStorage.java:1191)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2211)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:241)
	at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:171)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2758)
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:102)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1805)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1796)
	at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:63)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLInsert.saveRecord(OCommandExecutorSQLInsert.java:338)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLInsert.execute(OCommandExecutorSQLInsert.java:235)
	at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3208)
	at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3146)
	at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.command(ONetworkProtocolBinary.java:1529)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.executeRequest(ONetworkProtocolBinary.java:646)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.sessionRequest(ONetworkProtocolBinary.java:384)
	at com.orientechnologies.orient.server.network.protocol.binary.ONetworkProtocolBinary.execute(ONetworkProtocolBinary.java:207)
	at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)

Steps to reproduce

OPartitionedDatabasePool pool = new OPartitionedDatabasePool("remote:localhost/enterprise", "admin", "admin");
while（true）{				
long start = System.currentTimeMillis();
ODatabaseDocumentTx insertTx = pool.acquire();
ODatabaseRecordThreadLocal.INSTANCE.set(insertTx);
String sql = insertSql;
OCommandSQL insertSql = new OCommandSQL(sql);
insertSql.execute();
insertTx.commit();
insertTx.close();
}
return String.format("spend %s ms on insert orientdb table from hbase %s, total count is %s ",
						(System.currentTimeMillis() - start) , TABLE_NAME, SAVE_BATCH_COUNT);

batch insert sql like below with 2000 records in a sql:
INSERT INTO Company(rowkey,_key,name,business_status,ctime,entity_type,capital,city,enterprise_type,industry,operation_startdate,province,utime) VALUES ('09A626712730E39B779AB5BDA1E318D0','','','','','','','','','','','','2017-06-2718:01:37'),('09A626D27DB8C7C98678C3D8CB8737EF','09A626D27DB8C7C98678C3D8CB8737EF','','','2017-06-2718:02:04','Company','','','','','2012-08-13','',''),('09A626D27DB8C7C98678C3D8CB8737EF','','','','','','','','','','','','2017-06-2718:02:04'),('09A626EC2ED573F799C603CCC3244B2C','09A626EC2ED573F799C603CCC3244B2C','','','2017-06-2717:59:02','Company','100000.00','','','','2010-12-21','','')

run sometimes then always get the above exception

[zlfsimon]

after read the source code, it seems not reuse the allocated buffer and always to allocate new buffer

public ByteBuffer acquireDirect(boolean clear) {
    // check the pool first.
    final ByteBuffer buffer = pool.poll();

    if (buffer != null) {
      poolSize.decrementAndGet();

      if (clear) {
        buffer.position(0);
        buffer.put(new byte[pageSize]);
      }

      buffer.position(0);

      return trackBuffer(buffer);
    }
.........

Some way can I reuse the allocated buffer?

[zlfsimon]

my server.sh command result:
/usr/bin/java -d64 -server -XX:+UseG1GC -Xms2G -Xmx4G -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -XX:MaxDirectMemorySize=4g -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Djava.util.logging.config.file=/orientdb-community-importers-2.2.25/config/orientdb-server-log.properties -Dorientdb.config.file=/orientdb-community-importers-2.2.25/config/orientdb-server-config.xml -Dorientdb.www.path=/orientdb-community-importers-2.2.25/www -Dorientdb.build.number=2.2.x@r72a35a4bb42c4ca33272e7a59ad4c7a1d3ee859d; 2017-08-02 13:50:43+0000 -cp /orientdb-community-importers-2.2.25/lib/orientdb-server-2.2.25.jar:/orientdb-community-importers-2.2.25/lib/*:/orientdb-community-importers-2.2.25/plugins/* com.orientechnologies.orient.server.OServerMain

I know add XX:MaxDirectMemorySize can solve this issue, but it's still a critical risk: memory leak in produce env. Orientdb out of work if occur this issue.

[lvca]

XX:MaxDirectMemorySize is just an upper limit for the JVM, you can safely set it to a very high value, like 512G, because OrientDB would just use a little amount.

[andrii0lomakin]

@zlfsimon why do you think we do not reuse allcoated buffer , I can not get it . Could you elaborate ?

[andrii0lomakin]

@lvca I reopened issue because user thinks there is reas of memory leak, let's discuse with @zlfsimon his concerns

[andrii0lomakin]

@zlfsimon btw could you also check https://drive.google.com/file/d/0B2oZq2xVp841THlxbVhhemxMMGM/view?usp=sharing whether an exception is gone with this build?

[zlfsimon]

@lvca, I try to use XX:MaxDirectMemorySize=512G, but still get this exception, so I change this setting, from error stack, seems it's happen on allocate new buffer

[andrii0lomakin]

@zlfsimon what is the value of properties of JMX bean java.nio:type=BufferPool,name=direct after OOM? Could you check them? Could you also use a distribution which I provided to you?

[zlfsimon]

@Laa I use this distribution with same setting in server.sh, it works fine and no OOM happen, I also check the jvm heap, it seems better then the version 2.2.25 to use heap.
Thanks for your great help!

[andrii0lomakin]

@zlfsimon so we can treat this issue as resolved? Please do not close it I have to write release notes.

[zlfsimon]

@Laa yes, thanks

[andrii0lomakin]

@santo-it for release notes: "OrientDB able to automatically correct direct memory consumption even if flag XX:MaxDirectMemorySize is not set. But because OrientDB allocates direct memory by the big amount of chunks in previous versions amount of memory to be allocated in the last chunk can exceed direct memory limits. This issue was fixed." It is fixed under #7586 issue. The last issue is not confirmed yet but code already in the repository.