Inserting records with batch is slowing down after around 2000 records. #7652

HuangKBAaron · 2017-08-12T08:12:23Z

OrientDB Version: 2.2.25

Java Version: Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS: Windows 10 x64

HARDWARE: i5 (4 core), 8GB MEMORY, SSD

Actual behavior

The inserting is slowing down after 2000 records inserted with batch. With the 2000 records, the inserting process costs within 0.5 second, after that it becomes 3~4 seconds. If I kill the process and restart it again, the first 2000 record is as fast as the first time and slow down right after.

Steps to reproduce

sample script of one batch:
BEGIN;
LET vol=CREATE VERTEX v_volume SET f_year=2017,f_quantity=59,f_recorded='2017-08-08 15:58:44';
LET province=UPDATE v_province SET f_name='Guangdong', f_recorded='2017-08-08 15:58:44' UPSERT RETURN AFTER @Rid WHERE f_name='Guangdong';
LET city=UPDATE v_city SET f_name='Guangzhou', f_recorded='2017-08-08 15:58:44' UPSERT RETURN AFTER @Rid WHERE f_name='Guangzhou';
LET pr=UPDATE v_plan_round SET f_name='PR67', f_number=67, f_recorded='2017-08-08 15:58:44', f_updated='2017-08-08 15:58:44' UPSERT RETURN AFTER @Rid WHERE f_number=67;
LET brand=UPDATE v_brand SET f_name='Bentley', f_recorded='2017-08-08 15:58:44' UPSERT RETURN AFTER @Rid WHERE f_name='Bentley';
LET model=UPDATE v_model SET f_name='Continental GT/GTC/SSp', f_recorded='2017-08-08 15:58:44' UPSERT RETURN AFTER @Rid WHERE f_name='Continental GT/GTC/SSp';
CREATE EDGE e_in_pr FROM $vol TO $pr SET f_recorded='2017-08-08 15:58:44';
CREATE EDGE e_has_type FROM $vol TO $brand SET f_recorded='2017-08-08 15:58:44';
CREATE EDGE e_has_type FROM $vol TO $model SET f_recorded='2017-08-08 15:58:44';
CREATE EDGE e_has_type FROM $vol TO [#41:1,#47:0,#18:0,#22:0,#38:0,#29:0,#25:0] SET f_recorded='2017-08-08 15:58:44';
COMMIT RETRY 100;

local server config:

LOCAL SERVER CONFIGURATION
+----+-------------------------- |# |NAME +----+-------------------------- |0 |environment.dumpCfgAtStartup |1 |environment.concurrent |2 |environment.lockManager.co |3 |environment.allowJVMShutdown |4 |script.pool.maxSize |5 |memory.useUnsafe |6 |memory.chunk.size |7 |memory.directMemory.safeMode |8 |memory.directMemory.trackMode |9 |memory.directMemory.onlyAl |10 |jvm.gc.delayForOptimize |11 |storage.openFiles.limit |12 |storage.componentsLock.cache |13 |storage.diskCache.pinnedPages |14 |storage.diskCache.bufferSize |15 |storage.diskCache.writeCachePart |16 |storage.diskCache.writeCachePageTTL |17 |storage.diskCache.writeCac |18 |storage.diskCache.writeCac |19 |storage.diskCache.writeCac |20 |storage.diskCache.diskFree |21 |storage.diskCache.diskFree |22 |storage.diskCache.keepState |23 |storage.diskCache.checksumMode |24 |storage.configuration.syncOnUpdate |25 |storage.compressionMethod |26 |storage.encryptionMethod |27 |storage.encryptionKey |28 |storage.makeFullCheckpointAfterCreate |29 |storage.makeFullCheckpointAfterOpen |30 |storage.makeFullCheckpoint |31 |storage.trackChangedRecordsInWAL |32 |storage.useWAL |33 |storage.wal.syncOnPageFlush |34 |storage.wal.cacheSize |35 |storage.wal.fileAutoCloseInterval |36 |storage.wal.maxSegmentSize |37 |storage.wal.maxSize |38 |storage.wal.commitTimeout |39 |storage.wal.shutdownTimeout |40 |storage.wal.fuzzyCheckpointInterval |41 |storage.wal.reportAfterOpe |42 |storage.wal.restore.batchSize |43 |storage.wal.readCacheSize |44 |storage.wal.fuzzyCheckpointShutdownWait |45 |storage.wal.fullCheckpoint |46 |storage.wal.path |47 |storage.diskCache.pageSize |48 |storage.diskCache.diskFreeSpaceLimit |49 |storage.lowestFreeListBound |50 |storage.lockTimeout |51 |storage.record.lockTimeout |52 |storage.useTombstones |53 |record.downsizing.enabled |54 |object.saveOnlyDirty |55 |db.pool.min |56 |db.pool.max |57 |db.pool.idleTimeout |58 |db.pool.idleCheckDelay |59 |db.mvcc.throwfast |60 |db.validation |61 |nonTX.recordUpdate.synch |62 |nonTX.clusters.sync.immediately |63 |tx.trackAtomicOperations |64 |tx.pageCacheSize |65 |index.embeddedToSbtreeBonsaiThreshold |66 |index.sbtreeBonsaiToEmbeddedThreshold |67 |hashTable.slitBucketsBuffer.length |68 |index.auto.synchronousAutoRebuild |69 |index.auto.lazyUpdates |70 |index.flushAfterCreate |71 |index.manual.lazyUpdates |72 |index.durableInNonTxMode |73 |index.ignoreNullValuesDefault |74 |index.txMode |75 |index.cursor.prefetchSize |76 |sbtree.maxDepth |77 |sbtree.maxKeySize |78 |sbtree.maxEmbeddedValueSize |79 |sbtreebonsai.bucketSize |80 |sbtreebonsai.linkBagCache.size |81 |sbtreebonsai.linkBagCache.evictionSize |82 |sbtreebonsai.freeSpaceReuseTrigger |83 |ridBag.embeddedDefaultSize |84 |ridBag.embeddedToSbtreeBonsaiThreshold |85 |ridBag.sbtreeBonsaiToEmbed |86 |collections.preferSBTreeSet |87 |file.trackFileClose |88 |file.lock |89 |file.deleteDelay |90 |file.deleteRetry |91 |security.userPasswordSaltIterations |92 |security.userPasswordSaltCacheSize |93 |security.userPasswordDefaultAlgorithm |94 |network.maxConcurrentSessions |95 |network.socketBufferSize |96 |network.lockTimeout |97 |network.socketTimeout |98 |network.requestTimeout |99 |network.retry.strategy |100 |network.retry |101 |network.retryDelay |102 |network.binary.loadBalancing.enabled |103 |network.binary.loadBalancing.timeout |104 |network.binary.maxLength |105 |network.binary.readResponse.maxTimes |106 |network.binary.debug |107 |network.http.installDefaultCommands |108 |network.http.serverInfo |109 |network.http.maxLength |110 |network.http.streaming |111 |network.http.charset |112 |network.http.jsonResponseError |113 |network.http.jsonp |114 |network.http.sessionExpireTimeout |115 |network.http.useToken |116 |network.token.secretKey |117 |network.token.encryptionAlgorithm |118 |network.token.expireTimeout |119 |profiler.enabled |120 |profiler.config |121 |profiler.autoDump.interval |122 |profiler.maxValues |123 |profiler.memoryCheckInterval |124 |sequence.maxRetry |125 |sequence.retryDelay |126 |storageProfiler.intervalBe |127 |storageProfiler.cleanUpInterval |128 |log.console.level |129 |log.file.level |130 |class.minimumClusters |131 |log.console.ansi |132 |cache.local.impl |133 |command.timeout |134 |command.cache.enabled |135 |command.cache.evictStrategy |136 |command.cache.minExecutionTime |137 |command.cache.maxResultsetSize |138 |query.parallelAuto |139 |query.parallelMinimumRecords |140 |query.parallelResultQueueSize |141 |query.scanPrefetchPages |142 |query.scanBatchSize |143 |query.scanThresholdTip |144 |query.limitThresholdTip |145 |query.live.support |146 |query.timeout.defaultStrategy |147 |lucene.query.pageSize |148 |statement.cacheSize |149 |sql.graphConsistencyMode |150 |client.channel.maxPool |151 |client.connectionPool.waitTimeout |152 |client.channel.dbReleaseWaitTimeout |153 |client.ssl.enabled |154 |client.ssl.keyStore |155 |client.ssl.keyStorePass |156 |client.ssl.trustStore |157 |client.ssl.trustStorePass |158 |server.openAllDatabasesAtStartup |159 |server.channel.cleanDelay |160 |server.cache.staticFile |161 |server.log.dumpClientExceptionLevel |162 |server.log.dumpClientExcep |163 |distributed.dumpStatsEvery |164 |distributed.crudTaskTimeout |165 |distributed.maxStartupDelay |166 |distributed.commandTaskTimeout |167 |distributed.commandQuickTaskTimeout |168 |distributed.commandLongTaskTimeout |169 |distributed.deployDbTaskTimeout |170 |distributed.deployChunkTaskTimeout |171 |distributed.deployDbTaskCompression |172 |distributed.asynchQueueSize |173 |distributed.asynchResponsesTimeout |174 |distributed.purgeResponsesTimerDelay |175 |distributed.conflictResolv |176 |distributed.conflictResolv |177 |distributed.conflictResolv |178 |distributed.txAliveTimeout |179 |distributed.requestChannels |180 |distributed.responseChannels |181 |distributed.heartbeatTimeout |182 |distributed.checkHealthCanOfflineServer |183 |distributed.checkHealthEvery |184 |distributed.autoRemoveOfflineServers |185 |distributed.publishNodeStatusEvery |186 |distributed.localQueueSize |187 |distributed.dbWorkerThreads |188 |distributed.queueMaxSize |189 |distributed.backupDirectory |190 |distributed.backupTryIncrementalFirst |191 |distributed.concurrentTxMaxAutoRetry |192 |distributed.atomicLockTimeout |193 |distributed.concurrentTxAutoRetryDelay |194 |db.document.serializer |195 |client.krb5.config |196 |client.krb5.ccname |197 |client.krb5.ktname |198 |client.credentialinterceptor |199 |client.ci.keyalgorithm |200 |client.ci.ciphertransform |201 |client.ci.keystore.file |202 |client.ci.keystore.password |203 |security.createDefaultUsers |204 |server.security.file |205 |jna.disable.system.library |206 |distributed.queueTimeout |207 |db.makeFullCheckpointOnIndexChange |208 |db.makeFullCheckpointOnSchemaChange |209 |client.session.tokenBased |210 |oauth2.secretkey |211 |storage.cluster.usecrc32 |212 |lazyset.workOnStream |213 |db.mvcc |214 |db.use.distributedVersion |215 |mvrbtree.timeout |216 |mvrbtree.nodePageSize |217 |mvrbtree.loadFactor |218 |mvrbtree.optimizeThreshold |219 |mvrbtree.entryPoints |220 |mvrbtree.optimizeEntryPointsFactor |221 |mvrbtree.entryKeysInMemory |222 |mvrbtree.entryValuesInMemory |223 |mvrbtree.ridBinaryThreshold |224 |mvrbtree.ridNodePageSize |225 |mvrbtree.ridNodeSaveMemory |226 |tx.commit.synch |227 |tx.autoRetry |228 |tx.log.fileType |229 |tx.log.synch |230 |tx.useLog |231 |index.auto.rebuildAfterNotSoftClose |232 |client.channel.minPool |233 |storage.keepOpen |234 |cache.local.enabled +----+-------------------------- -------------------------+-------------------------------------------------------------------------+
|VALUE |
-------------------------+-------------------------------------------------------------------------+
|false |
|true |
ncurrency.level |32 |
|true |
|20 |
|true |
|2147483647 |
|true |
|false |
ignedMemoryAccess |true |
|600 |
|512 |
|10000 |
|20 |
|4096 |
|15 |
|86400 |
hePageFlushInterval |25 |
heFlushInactivityInterval|60000 |
heFlushLockTimeout |-1 |
SpaceCheckInterval |5 |
SpaceCheckIntervalInPages|2048 |
|true |
|StoreAndVerify |
|true |
|nothing |
|nothing |
| |
|false |
|true |
AfterClusterCreate |true |
|false |
|true |
|true |
|3000 |
|10 |
|128 |
|4096 |
|1000 |
|10000 |
|300 |
rationsDuringRestore |10000 |
|1000 |
|1000 |
|600 |
ShutdownTimeout |600 |
| |
|64 |
|256 |
|16 |
|0 |
|2000 |
|false |
|true |
|false |
|1 |
|100 |
|0 |
|0 |
|false |
|true |
|false |
|manindex |
|false |
|12 |
|40 |
|-1 |
|1500 |
|true |
|10000 |
|true |
|1 |
|true |
|false |
|FULL |
|500000 |
|64 |
|10240 |
|40960 |
|2 |
|100000 |
|1000 |
|0.5 |
|4 |
|40 |
dedToThreshold |-1 |
|false |
|false |
|true |
|10 |
|50 |
|65536 |
|500 |
|PBKDF2WithHmacSHA256 |
|1000 |
|0 |
|15000 |
|15000 |
|3600000 |
|auto |
|5 |
|500 |
|false |
|2000 |
|16384 |
|20 |
|false |
|true |
|OrientDB Server v.2.2.25 (build 72a35a4)|
|1000000 |
|false |
|utf-8 |
|true |
|false |
|300 |
|false |
| |
|HmacSHA256 |
|60 |
|false |
| |
|0 |
|200 |
|120000 |
|100 |
|200 |
tweenSnapshots |100 |
|5000 |
|info |
|info |
|0 |
|auto |
|com.orientechnologies.orient.core.cache.ORecordCacheWeakRefs |
|0 |
|false |
|PER_CLUSTER |
|10 |
|500 |
|false |
|300000 |
|20000 |
|20 |
|1000 |
|50000 |
|10000 |
|true |
|EXCEPTION |
|10000 |
|100 |
|tx |
|100 |
|5000 |
|10000 |
|false |
| |
| |
| |
| |
|false |
|5000 |
|false |
|FINE |
tionFullStackTrace |false |
|0 |
|10000 |
|10000 |
|120000 |
|5000 |
|86400000 |
|1200000 |
|60000 |
|7 |
|0 |
|15000 |
|15000 |
erRepairerChain |quorum,content,majority,version |
erRepairerCheckEvery |5000 |
erRepairerBatch |50 |
|30000 |
|1 |
|1 |
|10000 |
|false |
|10000 |
|0 |
|10000 |
|10000 |
|8 |
|10000 |
|../backup/databases |
|true |
|10 |
|50 |
|10 |
|ORecordSerializerBinary |
| |
| |
| |
| |
|AES |
|AES/CBC/PKCS5Padding |
| |
| |
|true |
| |
|true |
|500000 |
|true |
|true |
|true |
| |
|false |
|true |
|true |
|false |
|0 |
|256 |
|0.7 |
|100000 |
|64 |
|1.0 |
|false |
|false |
|-1 |
|64 |
|false |
|false |
|1 |
|classic |
|false |
|true |
|true |
|1 |
|true |
|true |
-------------------------+-------------------------------------------------------------------------+

Is there suggestion that i can get a work around? the process doesn't consume much memory (JAVA SE, consuming 1GB memory, nearly 40% CPU time). There are total 370,000 records.
I also try multi-processes, but it didn't work. The second/third process was even slower.

Also I changed some setting of the configure file:

    <properties>
        <entry value="4" name="db.pool.min"/>
        <entry value="64" name="db.pool.max"/>
        <entry value="true" name="profiler.enabled"/>
		<entry value="4294967294" name="memory.chunk.size"/>
		<entry value="42" name="storage.diskCache.pinnedPages"/>
		<entry value="262144" name="storage.diskCache.bufferSize"/>

After restart the server or restart java SE process. Those settings have got effect. Did I miss anything?

The text was updated successfully, but these errors were encountered:

luigidellaquila · 2017-08-14T08:01:40Z

Hi @HuangKBAaron

probably the slowdown is due to the lookups for UPSERT statemets, please make sure you have an index on the property you use in the WHERE condition of the UPSERT statements

Thanks

Luigi

HuangKBAaron · 2017-08-18T14:29:33Z

@luigidellaquila
I added index like the following:

CREATE INDEX i_segment_name ON v_segment (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_body_type_name ON v_body_type (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_fuel_type_name ON v_fuel_type (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_brand_indicator_name ON v_brand_indicator (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_ev_type_name ON v_ev_type (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_product_type_name ON v_product_type (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_city_level_name ON v_city_level (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_city_cluster_name ON v_city_cluster (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_sss_type_name ON v_sss_type (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_content_name ON v_content (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_model_name ON v_model (f_name) UNIQUE_HASH_INDEX STRING;
CREATE INDEX i_volume_row ON v_volume (f_row) NOTUNIQUE_HASH_INDEX Decimal;
CREATE INDEX i_plan_round_number ON v_plan_round (f_number) UNIQUE_HASH_INDEX Integer;
CREATE INDEX i_province_name ON v_province (f_name) NOTUNIQUE_HASH_INDEX STRING;
CREATE INDEX i_city_name ON v_city (f_name) NOTUNIQUE_HASH_INDEX STRING;
CREATE INDEX i_brand_name ON v_brand (f_name) UNIQUE_HASH_INDEX STRING;

Total affected record is 96. The problem remains. Is it something to do with the transaction?

lvca added question performance labels Aug 14, 2017

andrii0lomakin closed this as completed Aug 5, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inserting records with batch is slowing down after around 2000 records. #7652

Inserting records with batch is slowing down after around 2000 records. #7652

HuangKBAaron commented Aug 12, 2017 •

edited

Loading

luigidellaquila commented Aug 14, 2017

HuangKBAaron commented Aug 18, 2017

Inserting records with batch is slowing down after around 2000 records. #7652

Inserting records with batch is slowing down after around 2000 records. #7652

Comments

HuangKBAaron commented Aug 12, 2017 • edited Loading

OrientDB Version: 2.2.25

Java Version: Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

OS: Windows 10 x64

HARDWARE: i5 (4 core), 8GB MEMORY, SSD

Actual behavior

Steps to reproduce

local server config:

luigidellaquila commented Aug 14, 2017

HuangKBAaron commented Aug 18, 2017

HuangKBAaron commented Aug 12, 2017 •

edited

Loading