Write parquet files and Hive catalog support #1

thorfour · 2024-04-10T18:28:21Z

This adds

Can't use prefixed buckets because file paths are absolute

This uses Go's templating to correctly template in the partition fields in the Avro entry v1 schema.

Pass in interface instead of struct

thorfour added 21 commits April 4, 2024 10:46

v1 metadta builder

a05d208

make schema for v1 metadata a pointer so it can be copied in

f41d819

metadata v1 builder test

267e54c

go mod tidy

5521a08

avro col map helper function

f23eec1

hive catalog stubs

267d121

manifest builder

95cf0a6

remove io pkg

6938c9e

replace io.FS with objstore.Bucket

d12facd

fix unit tests for objstore migration

5ded5a3

hive catalog

335a68b

hive catalog

f2fec16

fix for bucket prefixes

c3892ab

Can't use prefixed buckets because file paths are absolute

adding manifest writing support

cb39355

Generate avro schema from entries

81fb819

This uses Go's templating to correctly template in the partition fields in the Avro entry v1 schema.

test: use the new WriteManifestV1 function

e59cd78

build data file from parquet file

d41cf0a

fixed manifest building to not copy locks from datafile

085736a

Pass in interface instead of struct

move avro schemas to top-level consts

b1eea2e

removed internal avro schema pkg

6fdedbc

added manifest list writer

d565dab

thorfour merged commit a754ea5 into main Apr 10, 2024
1 check passed

thorfour deleted the metadata-builder branch April 10, 2024 18:29

Provide feedback