enhancement (store): Add transaction management for SPARQL updates over HTTP

[benjamingeer]

This pull request adds application-level transaction management for SPARQL updates over HTTP. This is necessary to allow GraphDB to check updates for consistency, as mentioned in issue 33.

GraphDB does its consistency checks when a triplestore transaction is committed. Each HTTP request to the triplestore is one triplestore transaction. But when we create a resource with its values, we do an update to create an empty resource, and then an update for each value. Consistency constraints will be satisfied only when all the updates are finished. To make all those updates happen in one triplestore transaction, the transaction manager added here collects SPARQL updates in memory, and when it is asked to commit an (application-level) update transaction, it concatenates them so they're sent to the triplestore in a single (triplestore) transaction. See the documentation added to store-module.rst and triplestore-updates.rst for details.

Please look closely at ResourcesResponderV1.createNewResource, which I had to refactor a bit. It now creates the resource and the values in an (application-level) update transaction. Once that transaction is committed, it verifies that the resource and the values were created.

[tobiasschweizer]

• In knora-api-server/design-documentation/store-module.html, the term rapier should be replaced by Knora.
• triplestore updates -> Application-Level Locking: Are these supposed to be single quotes: ‘read committed’?

[benjamingeer]

After our Skype conversation just now, I changed UpdateNotPerformedException to extend InternalServerException, because it really shouldn't happen unless there's a bug. When I initially added it, I hadn't yet added resource locks, so it was possible that pre-update checks could succeed but the update could fail, because someone changed something in the triplestore between the two operations. But with resource locks, this shouldn't be possible.

In knora-api-server/design-documentation/store-module.html, the term rapier should be replaced by Knora.

Done.

triplestore updates -> Application-Level Locking: Are these supposed to be single quotes: ‘read committed’?

Yes, just to indicate that it's a technical term.

[benjamingeer]

Just so there's a record of this: We considered the possibility that create-resource-with-values might partially succeed (because of a bug in Knora), e.g. create the resource with one missing value. If GraphDB is doing consistency checks, in the worst case, the missing value wouldn't be a required value, because otherwise the whole transaction would be rolled back. So at worst, the user would get a resource with a missing optional value (meaning that database consistency would be maintained), and they'd get an error message asking them to report it as a possible bug (the error message would also be logged). We considered whether the application should then try to undo the whole operation by deleting the resource and the values that were created, but I argued that if there's a bug, it would be better to preserve the evidence.

[tobiasschweizer]

In any case, the GUI should be able to handle an error message properly. At the moment, the error message returned by the server (as JSON) is just displayed to the user as a JavaScript alert (the user may have no idea what that means). We should handle that in a more user friendly way and we as administrators should also be notified by the logging you mentioned.

[benjamingeer]

Maybe we should redesign ApiStatusCodesV1 to have status codes that are actually relevant to Knora. If the GUI isn't actually doing anything with the existing status codes, we could get rid of the ones that are no longer relevant, and add new ones to represent the different exceptions Knora can throw. Then we could change the GUI to do something useful depending on the error code it receives.

[tobiasschweizer]

In ResourcesResponderV1 on line 1031ff., could you put a comment explaining that here actually a lambda function is passed to TransactionUtil.runInUpdateTransaction? You did document this already in the scaladoc of runInUpdateTransaction for the first param, but probably this makes it more clear.

Could also please add some documentation to rst Transaction Management how the SPARQL messages are stored to a Map using the unique transaction id and then finally concatenated (addUpdateToTransaction and concatenateAndForgetUpdates)? You explained well why we need transactions but the doumentation of its actual implementation is rather brief in rst.

[tobiasschweizer]

Speaking about logging, I think that any Exception of type InternalServerException should be reported to us because this is not the user's fault.

Probably we do that already (Exceptions.scala):

1. Exceptions extending InternalServerException mean that the problem is not the client's fault. They are
   reported to the client and are also logged (e.g. to the console). When thrown in an actor, they are also
   escalated to the actor's supervisor.

[benjamingeer]

Probably we do that already

Yes. :) Knora's exception hierarchy has a root, KnoraException, with two branches: RequestRejectedException and InternalServerException. Exception logging happens in two places: in ActorUtil.future2Message and in RouteUtilV1. In both places, any exception that doesn't extend RequestRejectedException is logged.

[benjamingeer]

In ResourcesResponderV1 on line 1031ff., could you put a comment explaining that here actually a lambda function is passed to TransactionUtil.runInUpdateTransaction?

Done.

Could also please add some documentation to rst Transaction Management how the SPARQL messages are stored to a Map using the unique transaction id and then finally concatenated (addUpdateToTransaction and concatenateAndForgetUpdates)?

Done.

[tobiasschweizer]

Great!

Now this is much easier to understand when reading the documentation.

When do actually intend to activate GraphDB's consistency checker? And where would that happen? Somewhere in the ttl-files we use to create the test rep?

[tobiasschweizer]

I think you can merge the branch to develop.

[benjamingeer]

When do actually intend to activate GraphDB's consistency checker? And where would that happen? Somewhere in the ttl-files we use to create the test rep?

I'm going to make a branch tomorrow for this. Yes, you have to activate it when you create the repository. You can see how it's done in issue 33. Since it requires the reasoner, I'll have to change the SPARQL update templates first so they'll work with reasoning. Then I can try turning on the consistency checker and see what happens.

I think you can merge the branch to develop.

I'm just going to change one more thing first.

[benjamingeer]

I refactored one thing: I gave the store module responsibility for making transaction IDs, instead of generating them in TransactionUtil. I think this makes more sense, because if we switched to using a triplestore's native transaction API, the store module would probably have to get the transaction IDs from that API.