Avoid writing NaN and Infinity with json format table #24558
Conversation
| @@ -123,7 +123,16 @@ else if (REAL.equals(type)) { | |||
| return (generator, block, position) -> generator.writeNumber(REAL.getFloat(block, position)); | |||
| } | |||
| else if (DOUBLE.equals(type)) { | |||
| return (generator, block, position) -> generator.writeNumber(DOUBLE.getDouble(block, position)); | |||
| return (generator, block, position) -> { | |||
There was a problem hiding this comment.
'NaN' and 'Infinity' getting casted as Double it seems. It goes through double datatype.
There was a problem hiding this comment.
I don't think so. Please try CREATE TABLE test_nan WITH (format = 'json') AS SELECT real 'NaN' a;.
sug-ghosh
force-pushed
the
invalid_json_write
branch
from
360d7a0 to
a0b1253
Compare
sug-ghosh
force-pushed
the
invalid_json_write
branch
from
a0b1253 to
bc3afbe
Compare
| if (!Float.isNaN(value) && !Float.isInfinite(value)) { | ||
| generator.writeNumber(value); | ||
| } |
There was a problem hiding this comment.
Float.isFinite() will cover both infinite and NaN, as the comparison of NaN with any Float (MAX_VALUE in this case) will return false
Same apply to Double type handling below
| @@ -120,10 +122,26 @@ else if (type instanceof DecimalType decimalType) { | |||
| }; | |||
| } | |||
| else if (REAL.equals(type)) { | |||
There was a problem hiding this comment.
Similar logic is also in createMapKeyFunction(Type type)
There was a problem hiding this comment.
even though they will not break as these, but still worth discussing on how those values supposed to be read back especially considering cases like duplicate map keys of NaN etc., or just leave them as is and that the downstream consumer decide
There was a problem hiding this comment.
In other cases, we've preferred to emulate the Hive SerDe behavior. What happens in Hive if you serialize maps with NaN / Infinite / -Infinite keys there, @sug-ghosh ?
| generator.writeNumber(value); | ||
| } | ||
| else { | ||
| throw new TrinoException(StandardErrorCode.INVALID_JSON_LITERAL, "Invalid value to Insert Real " + value); |
There was a problem hiding this comment.
would prefer to add : to separate the value out more obviously
fixes #20395 #20395
This pr will fix inserting NaN and Infinity into table as Json Format.
In hive, if we insert NaN or Infinity in json format table, it would store that as NULL, which readable from Trino. We can go that path also if we want to create table to be successful.
@ebyhr @hashhar
Description
Additional context and related issues
Release notes
( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text: