Code

feat(ingest): Glue jobs#2687

Code

Merged

shirshanka merged 60 commits intodatahub-project:masterdatahub-project/datahub:masterfrom kevinhu:glue-etlkevinhu/datahub:glue-etlCopy head branch name to clipboardJun 22, 2021

+2,248-199

Commits

Commits on Jun 9, 2021

• Update README

  kevinhu

  committed
  334a485

  Copy full SHA for 334a485

• Merge branch 'master' of github.com:kevinhu/datahub into glue-etl

  kevinhu

  committed
  8580c54

  Copy full SHA for 8580c54

Commits on Jun 10, 2021

• Merge branch 'linkedin:master' into glue-etl

  kevinhu

  committed
  499d308

  Copy full SHA for 499d308

• Merge branch 'glue-etl' of github.com:kevinhu/datahub into glue-etl

  kevinhu

  committed
  520b099

  Copy full SHA for 520b099

• Merge branch 'linkedin:master' into glue-etl

  kevinhu

  committed
  1941e93

  Copy full SHA for 1941e93

Commits on Jun 11, 2021

• Read transformation DAGs

  kevinhu

  committed
  5270fac

  Copy full SHA for 5270fac

• Extract node sources

  kevinhu

  committed
  67c0807

  Copy full SHA for 67c0807

• Init glue MCEs

  kevinhu

  committed
  3ffdb1a

  Copy full SHA for 3ffdb1a

• Refactor job and flow wus

  kevinhu

  committed
  bdae7c1

  Copy full SHA for bdae7c1

• Resolve source and sink datasets

  kevinhu

  committed
  4acc825

  Copy full SHA for 4acc825

• Merge branch 'linkedin:master' into glue-etl

  kevinhu

  committed
  68ed8e5

  Copy full SHA for 68ed8e5

• Set URNs correctly

  kevinhu

  committed
  2fd692b

  Copy full SHA for 2fd692b

• Isort and update snapshot JSONs

  kevinhu

  committed
  a0bc357

  Copy full SHA for a0bc357

• Successful ingestion

  kevinhu

  committed
  e4b4d64

  Copy full SHA for e4b4d64

• Refactor job listing

  kevinhu

  committed
  7f0eb42

  Copy full SHA for 7f0eb42

• Glue ETL comments

  kevinhu

  committed
  fe63ce6

  Copy full SHA for fe63ce6

• Clean up s3 naming

  kevinhu

  committed
  218338e

  Copy full SHA for 218338e

Commits on Jun 12, 2021

• Add job properties

  kevinhu

  committed
  5c1c9f2

  Copy full SHA for 5c1c9f2

Commits on Jun 14, 2021

• Fix lint errors

  kevinhu

  committed
  5e4873a

  Copy full SHA for 5e4873a

• Temp disable extract_transform in tests

  kevinhu

  committed
  4739367

  Copy full SHA for 4739367

• Fix S3 URN

  kevinhu

  committed
  ce58f7b

  Copy full SHA for ce58f7b

• Stubs for S3

  kevinhu

  committed
  a37b1db

  Copy full SHA for a37b1db

• Fix lint errors

  kevinhu

  committed
  5050b8f

  Copy full SHA for 5050b8f

Commits on Jun 15, 2021

• Create Glue golden MCE json

  kevinhu

  committed
  ff682a8

  Copy full SHA for ff682a8

• Trim Glue golden MCE

  kevinhu

  committed
  5144ffc

  Copy full SHA for 5144ffc

• Reapply freeze to Glue files

  kevinhu

  committed
  ab419a4

  Copy full SHA for ab419a4

• Fix golden path

  kevinhu

  committed
  ecd89a6

  Copy full SHA for ecd89a6

• Merge

  kevinhu

  committed
  06d6c8a

  Copy full SHA for 06d6c8a

• Fix duplicate MCEs

  kevinhu

  committed
  51ae22a

  Copy full SHA for 51ae22a

• Fix outputDatasets

  kevinhu

  committed
  108a203

  Copy full SHA for 108a203

• Remove S3 URIs

  kevinhu

  committed
  d9d8bc5

  Copy full SHA for d9d8bc5

• Expand job names

  kevinhu

  committed
  e4a96b4

  Copy full SHA for e4a96b4

• Expand job custom props

  kevinhu

  committed
  db9d8ca

  Copy full SHA for db9d8ca

• Update golden

  kevinhu

  committed
  7e6333b

  Copy full SHA for 7e6333b

• Remove ownership classes

  kevinhu

  committed
  4026917

  Copy full SHA for 4026917

• Clean up redundant properties

  kevinhu

  committed
  c436149

  Copy full SHA for c436149

• Merge branch 'linkedin:master' into glue-etl

  kevinhu

  committed
  8619918

  Copy full SHA for 8619918

• Fix topological sort

  kevinhu

  committed
  33dd0ea

  Copy full SHA for 33dd0ea

• Fix S3 browse paths

  kevinhu

  committed
  e0eaf0c

  Copy full SHA for e0eaf0c

• Restore feast

  kevinhu

  committed
  6f7b74a

  Copy full SHA for 6f7b74a

• Smaller stubs

  kevinhu

  committed
  c738a74

  Copy full SHA for c738a74

Commits on Jun 16, 2021

• Update README

  kevinhu

  committed
  1d89927

  Copy full SHA for 1d89927

Commits on Jun 17, 2021

• Resolve golden script conflict

  kevinhu

  committed
  fe5b087

  Copy full SHA for fe5b087

• Regenerate snapshot JSONs

  kevinhu

  committed
  948dd24

  Copy full SHA for 948dd24

• Merge

  kevinhu

  committed
  8a681cd

  Copy full SHA for 8a681cd

• Rebuild

  kevinhu

  committed
  95efa7e

  Copy full SHA for 95efa7e

• Refactor node processing

  kevinhu

  committed
  55e6619

  Copy full SHA for 55e6619

• Add links to boto docs

  kevinhu

  committed
  51bdb88

  Copy full SHA for 51bdb88

Commits on Jun 18, 2021

• Fix sequence type error

  kevinhu

  committed
  59bbf15

  Copy full SHA for 59bbf15

• Fix Id typo

  kevinhu

  committed
  2a4501b

  Copy full SHA for 2a4501b

• Merge branch 'linkedin:master' into glue-etl

  kevinhu

  authored
  8feb874

  Copy full SHA for 8feb874

• Types for process_dataflow_graph

  kevinhu

  committed
  d173d6c

  Copy full SHA for d173d6c

• Include extension type in glue imports

  kevinhu

  committed
  1b44baf

  Copy full SHA for 1b44baf

Commits on Jun 19, 2021

• S3 deduplication logic

  kevinhu

  committed
  d80f529

  Copy full SHA for d80f529

• Fix type annotation

  kevinhu

  committed
  db1c638

  Copy full SHA for db1c638

• Add comments for deduplication

  kevinhu

  committed
  8eb3d50

  Copy full SHA for 8eb3d50

Commits on Jun 21, 2021

• Fix dataset IDs for Glue

  kevinhu

  committed
  5b66074

  Copy full SHA for 5b66074

• Merge branch 'linkedin:master' into glue-etl

  kevinhu

  authored
  9955e06

  Copy full SHA for 9955e06

• Update golden files

  kevinhu

  committed
  6335f95

  Copy full SHA for 6335f95

Commits on Jun 22, 2021

• Merge

  kevinhu

  committed
  09b9b60

  Copy full SHA for 09b9b60