asynchronous database performance analysis

[synctext]

We now have blocking database IO. All database interaction is within the main twisted thread. It blocks, causing performance loss.

Events can be split in multiple parts. Events would continue when data has been retrieved, Twisted feature: https://twistedmatrix.com/documents/8.1.0/api/twisted.internet.defer.Deferred.html

Also, Lipu has an experimental branch using storm framework.

Task: benchmarking. Compare performance of various approaches, without full implementations (takes too much dev weeks).

[lfdversluis]

See #2137 :)

[synctext]

As discussed today:

• study Execute and ExecuteMany in Dispersy and Tribler
• Create stress test job
• serial IO thread in threadpool with inline deferred calls.

[synctext]

Work by @qstokkink
Tribler/dispersy@devel...qstokkink:threadeddb

This 2 weeks sprint contains operational multi-core Dispersy using Twisted.
40x speedup of sync experiment with 10 peers.

ThreadSafeExecute of Dispersy database access. Database requests become serialized. However, they can be requested from multiple Twisted threads! Tribler/dispersy@devel...qstokkink:threadeddb#diff-0d31758bc4ed14da7f82a5780aa66abdR38

Most of the speedup comes from a multi-core version of the Dispersy endpoint.py using Twisted: Tribler/dispersy@devel...qstokkink:threadeddb#diff-a694510bf8dca13af2bd89656ea772e0R210

[lfdversluis]

@synctext the reason this approach works is that it does not return a database item from _deep_copy_nts_db_call. it only returns the last inserted id or an iterator for the list generated by fetchall. The moment this returns a cursor it explodes :D Having said that, it looks well implemented in terms of synchronous code by making use of blockingCallFromThread, but note that this will block the reactor thread. http://twistedmatrix.com/documents/current/api/twisted.internet.threads.blockingCallFromThread.html

[synctext]

Update:
We aim to revisit this mega-task. After replacing Dispersy, deploying Giga channels, and Libtorrent wrapper re-write this is scheduled next.

[ichorid]

Closed, as we should instead talk about particular databases, like Channels, or TC, etc.