Logical datatype support in spark-avro

[rahulmittal91]

Hi ,

Is there a functionality here which can deserialize data to support reflect data apis.

For instance if my type schema for field is

   {"name": "a", "type": "string" , "java-class": "java.math.BigInteger"},

I want the return type to be of BigInteger type .

[JoshRosen]

Nope, it looks like we don't support this currently. I'd be happy to review a PR to add support for this, though.

[rahulmittal91]

We have a pull request
#121
which supports LogicalData types in avro. Currently we have added support for BigDecimals with scale and precision , timestamps and date .
This code is also running in our production from past 8 months without any issues .

[JoshRosen]

To clarify, it looks like #121 is concerned with logicalType, whereas it sounds like this issue is discussing ReflectData / the java-class attribute. These are related but slightly orthogonal features, right?

[rahulmittal91]

Yes these are exaclty not the same features . We came up with a design where logical types were more suitable . We wanted to represent db types(Bigdecimals, dates and timestamps) in avro types, serialize and deserialize them (This question was posted before #121 was raised).

[cpbhagtani]

@JoshRosen, me and Rahul and the contributors to this PR. This issues was raised as a question if there was such support in spark-avro. Since there was no response, so we implemented this feature. Initially we thought of doing it through ReflectData, but logical types suited best to our use case.

@rahulmittal91 Can you change the subject of this issue?