Support for logical datatype like decimal type

[varsha-abhinandan]

This pull request is in context to - #121 which was done by @cpbhagtani to support the logical types. This current PR adds this support for spark 2.x

[codecov-io]

Codecov Report

Merging #257 into master will increase coverage by 0.61%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #257      +/-   ##
==========================================
+ Coverage   90.77%   91.38%   +0.61%     
==========================================
  Files           5        5              
  Lines         336      360      +24     
  Branches       50       55       +5     
==========================================
+ Hits          305      329      +24     
  Misses         31       31

[gengliangwang]

In Avro spec:

A decimal logical type annotates Avro bytes or fixed types

Can you handle bytes and fixed type as well?

[cbia4]

Is there any movement on this? I have yet to find a way to load avro data into a data frame where data with type bytes (and logical type decimal with scale and precision) is converted back to a human readable form.

[cbia4]

I updated the avro package version and rebuilt SchemaConverters.scala with something similar to what is in the PR - except like I mentioned, Avro 1.8.2 stores the decimal logical type as bytes, and you need to use Conversions.DecimalConversion from the avro API to make it happen, but it works as expected for anyone who needs to get this to work with decimal types.

[peter-ungvari]

@cbia4 Where can I find your changes?
Is there another PR for bytes avro type?

I am ready to work on this in order to make spark-avro compatible with avro spec as well as Hive regarding to decimal types.

[LuboVarga]

I wish this project is in maven... I am more familiar with it. But I can promise to share result of my work (decimal type for bytes and fixed types and perhaps timestamp logical type)

[koertkuipers]

sbt has a funny syntax but on the console it pretty easy to use as a developer. its compile and test cycle is much faster than maven. you will be happy with all the time you get back not spending waiting on super slow maven:)

…

On Fri, Jun 8, 2018 at 2:32 AM, LuboVarga ***@***.***> wrote: I wish this project is in maven... I am more familiar with it. But I can promise to share result of my work (decimal type for bytes and fixed types and perhaps timestamp logical type) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#257 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAyIJF3Bs8MyM8UzgHAjSWKh2MgWnoasks5t6hpsgaJpZM4QrWv5> .