Refactor!(hive): improve transpilation of TO_JSON #1809
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related discussion: #1803 (comment)
This is a best-effort approach, so it might not be complete. The difficulty of transpiling these JSON calls correctly stems from the fact that Hive / Spark work with JSON values using only their string representation (*), whereas Trino and Presto have native support for them, using a
JSON
type.Note: all changed tests were previously failing in Spark. The new ones work and have the same semantics as in Trino.
(*) Or by parsing them, e.g. with
FROM_JSON
, and converting them into native struct types likestruct
andmap
.References: