ISSUE-509: Provide a delete API for schema metadata #510
Conversation
| if (schemaVersionInfos != null) { | ||
| for (SchemaVersionInfo schemaVersionInfo: schemaVersionInfos) { | ||
| invalidateCachesAndNotifyAllHAServers(schemaVersionInfo); | ||
| schemaMetadataId = schemaVersionInfo.getSchemaMetadataId(); |
There was a problem hiding this comment.
Here, instead of invalidating one schema version at a time. I think we should expose a separate endpoint to invalidate schema metadata as a whole. It possible that during the process of invalidation there could be network outage, we might end up just invalidating a subset of versions leaving the schema metadata in a partial delete state.
There was a problem hiding this comment.
@raju-saravanan Sorry, forgot to reply to your comment. That's a good point and it's cleaner. Can we do it as part of a separate PR and merge this in if everything looks good? Reason is that we are anyways retrying notifying all HA servers at https://github.com/hortonworks/registry/blob/master/schema-registry/core/src/main/java/com/hortonworks/registries/schemaregistry/HAServerNotificationManager.java#L60 and even putting the current thread to sleep for 100 ms every time. So the delete call just won't return. And a network outage and out of sync could happen even if we provide a notifyHAServer at schema metadata level as well.
There was a problem hiding this comment.
@priyank5485: Agreed, but by making a single call to the each and every HA server, the call could either fail or succeed, depending upon the network connectivity, in other words, it has a transaction like property. The cache would reflect the old snapshot of database if network call didn't go through, or latest snapshot of the database if the network call went through but it would never reflect an intermediate state. This is approach would be a minor improvement, so we can take it up in a different PR.
| if (schemaVersionInfos != null) { | ||
| for (SchemaVersionInfo schemaVersionInfo: schemaVersionInfos) { | ||
| invalidateCachesAndNotifyAllHAServers(schemaVersionInfo); | ||
| schemaMetadataId = schemaVersionInfo.getSchemaMetadataId(); |
There was a problem hiding this comment.
@priyank5485: Agreed, but by making a single call to the each and every HA server, the call could either fail or succeed, depending upon the network connectivity, in other words, it has a transaction like property. The cache would reflect the old snapshot of database if network call didn't go through, or latest snapshot of the database if the network call went through but it would never reflect an intermediate state. This is approach would be a minor improvement, so we can take it up in a different PR.
priyank5485
force-pushed
the
ISSUE-509
branch
from
35fa8f1 to
dd477bb
Compare
|
@raju-saravanan I tried to add test cases for delete api in BasicSchemaRegistryClientOpsTest. I added the deleteSchema call at the end of doTestSchemaOps and after that i put some assertions to make sure no versions, branches, states, etc are returned for that schema using the get* methods on SR client. The issue i ran in to is that on server side deleteSchema only deletes schema metadata row and relies on cascade delete to clean up all other tables. However, the test case uses in memory store which does not do that. As a result when i ask for all versions for that schema using getAllVersions call after deletion, that goes through a different route which does not check for schema metadata entry. It goes by branch and tries to get all versions for that branch. The approach is similar for a bunch of other server side methods as well. If you have any suggestions to quickly fix this please let me know. Otherwise i will suggest merging this. I have already tested using the unsuccessful test cases that i added locally and made sure that SR client calling deleteSchema calls the server side api right. It's just that the assertions for different gets after that are not successful. I have also tested the server side execution using UI and mysql locally. |
|
@priyank5485: I think we need to tweak the in-memory DB to support cascade delete operations. We can come back to the test later, I will raise an issue for this, meanwhile, I will merge this PR. Thanks @priyank5485. |
No description provided.