Skip to content
This repository has been archived by the owner on Dec 20, 2018. It is now read-only.

TimeUnit conversion for BigQuery compatibility #271

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

devlucasc
Copy link

These changes allow to set option that make conversion of timestamp in long format using a specified precision, for example, microseconds.
The setting of precision option can make avro compatible with BigQuery format, that reads timestamp in microseconds.

These changes allow to set option that make conversion of timestamp in long format using a specified precision, for example, microseconds.
The setting of precision option can make avro compatible with BigQuery format, that reads timestamp in microseconds.
@codecov-io
Copy link

codecov-io commented Mar 4, 2018

Codecov Report

Merging #271 into master will increase coverage by 0.09%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #271      +/-   ##
=========================================
+ Coverage   92.21%   92.3%   +0.09%     
=========================================
  Files           5       5              
  Lines         321     325       +4     
  Branches       43      48       +5     
=========================================
+ Hits          296     300       +4     
  Misses         25      25

@gengliangwang
Copy link
Contributor

Hi @devlucasc ,
In my opinion:

  1. If we should add this option, we need to add it for all the data sources, instead of just for AVRO.
  2. We can use .map or SQL statement to change the output timestamp unit, which is simple and straight forward to me.

@devlucasc
Copy link
Author

Hi @gengliangwang

  1. The problem only occur in avro data sources
  2. Map is not good when has a complex structure as arrays, structs or array of structs that contains a timestamp field. In this case is needed to make map, flatmap and explode to treat.

Kind regards.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants