fix(ingest): correctly handle transformer patch semantics #6505
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ingestion
jjoyce0510
approved these changes
Nov 22, 2022
cccs-Dustin
pushed a commit
to CybercentreCanada/datahub
that referenced
this pull request
Feb 1, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The ownership transformer contains a clever optimization, where if it is set to PATCH and does not need to make any changes, no aspect is generated. The surrounding code did not handle this logic correctly, and would issue an upsert with the original ownership object, hence overwriting anything that lived on the server. This bug would only be triggered if the server already contained a superset of what the transformer was trying to add.
Many other dataset transformers also had a similar, subtle bug. For example, if using the
pattern_add_dataset_tagstransformer and a no new tags applied to a given dataset, the server's tags would get overwritten even if PATCH was specified. This PR should also resolve that class of bugs.Tested manually with the ownership, props, and tags transformers.
Checklist