feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ #6204

hsheth2 · 2022-10-14T05:34:55Z

Closes #4809 and closes #6132 and fixes #6145.

Checklist

The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
Links to related issues (if applicable)
Tests for the changes have been added/updated (if applicable)
Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

github-actions · 2022-10-14T05:56:38Z

Unit Test Results (metadata ingestion)

      8 files ±0       8 suites ±0 1h 0m 8s ⏱️ -18s
  757 tests ±0   754 ✔️ ±0 3 💤 ±0 0 ❌ ±0
1 516 runs ±0 1 509 ✔️ ±0 7 💤 ±0 0 ❌ ±0

Results for commit 6e91c86. ± Comparison against base commit ae2ea52.

♻️ This comment has been updated with latest results.

github-actions · 2022-10-14T06:00:01Z

Unit Test Results (build & test)

613 tests ±0 609 ✔️ ±0 11m 41s ⏱️ -17s
151 suites ±0     4 💤 ±0
151 files ±0     0 ❌ ±0

Results for commit 6e91c86. ± Comparison against base commit ae2ea52.

♻️ This comment has been updated with latest results.

.github/workflows/metadata-ingestion.yml

vincentkoc

Need to make one minor change

tommy-watts-depop · 2022-10-25T16:17:21Z

Hey, Is there an ETA for this?

hsheth2 · 2022-10-27T00:13:16Z

@tommy-watts-depop Can't commit to an ETA, but I'm hoping to wrap this up mid-next week

Vimpy · 2022-11-01T17:01:20Z

Hey any ETA to merge?

update pytest-docker feat(ingest): loosen sqlalchemy dep ci matrix update airflow stuff to match more airflow fixes always install the right sqlalchemy stubs sqlalchemy compat type fixes fix ci fix types in oracle fix more stuff with lint fix attrs typing more airflow type annotations refactor iolet preprocessing more type fixes even more type fixes markupsafe compat fix airflow tests more airflow compat tweak workflow fix imports skip dag bag load test fix sqlalchemy mypy final type fixes ignore more mypy issues add assert to handle mypy issue fix athena test modify airflow plugin

KulykDmytro · 2022-11-08T20:29:13Z

in meanwhile airflow released 2.4.2 version

Vimpy · 2022-11-09T09:33:05Z

Hey any ETA to merge this?

vincentkoc

LGTM

vincentkoc · 2022-11-10T08:14:42Z

@Vimpy needs another reviewer to approve the PR

vincentkoc · 2022-11-10T21:01:36Z

Lint tests failing:

ERROR: /home/runner/work/datahub/datahub/metadata-ingestion/src/datahub_provider/_plugin.py Imports are incorrectly sorted and/or formatted.
--- /home/runner/work/datahub/datahub/metadata-ingestion/src/datahub_provider/_plugin.py:before	2022-11-10 07:44:16.998670
+++ /home/runner/work/datahub/datahub/metadata-ingestion/src/datahub_provider/_plugin.py:after	2022-11-10 07:52:03.617106
ERROR: /home/runner/work/datahub/datahub/metadata-ingestion/src/datahub_provider/_lineage_core.py Imports are incorrectly sorted and/or formatted.

Also smoke tests failing, @hsheth2 can you clean up your commits?

keerthiis · 2022-11-11T13:24:25Z

@hsheth2 - we see this issue with traitlets & pyspark & grpcio too :
acryl-datahub[redshift] 0.8.45 depends on traitlets<5.2.2; extra == "redshift"
The user requested (constraint file for Airflow 2.3.2) traitlets==5.2.2.post1

acryl-datahub[airflow,datahub-business-glossary,datahub-kafka,datahub-rest,druid,glue,kafka,s3,sagemaker] 0.8.41 depends on pyspark==3.0.3; extra == "s3"
The user requested (constraint) pyspark==3.2.1

 acryl-datahub[airflow,datahub-business-glossary,datahub-kafka,datahub-rest,druid,glue,kafka,sagemaker] 0.8.45 depends on grpcio==1.44.0; extra == "kafka"
The user requested (constraint) grpcio==1.46.3

This reverts commit 4196e71.

treff7es

lgtm

hsheth2 · 2022-11-11T18:22:25Z

@keerthiis I'd recommend using the PythonVirtualenvOperator in your airflow setup - your root airflow image should only have acryl-datahub[airflow] or acryl-datahub-airflow-plugin (depending on if you're using the lineage backend or the plugin), and then the specific tasks would depend on acryl-datahub[<source-name>]

I'm planning on adding an example to the docs as well

Related to datahub-project#6204.

…nt (#6418) Related to #6204.

…roject#6204) Co-authored-by: Tamas Nemeth <[email protected]>

…nt (datahub-project#6418) Related to datahub-project#6204.

github-actions bot added devops PR or Issue related to DataHub backend & deployment ingestion PR or Issue related to the ingestion of metadata labels Oct 14, 2022

hsheth2 force-pushed the loosen-airflow branch from 23f99e2 to 8e381fb Compare October 14, 2022 07:12

hsheth2 mentioned this pull request Oct 14, 2022

Python package is incompatibility with Airflow 2.3.0 #4809

Closed

vincentkoc reviewed Oct 14, 2022

View reviewed changes

.github/workflows/metadata-ingestion.yml Outdated Show resolved Hide resolved

vincentkoc suggested changes Oct 14, 2022

View reviewed changes

hsheth2 changed the title ~~feat(ingest): loosen sqlalchemy dep~~ feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ Oct 25, 2022

hsheth2 changed the title ~~feat(ingest): loosen sqlalchemy dep & support airflow 2.3+~~ feat(ingest): loosen sqlalchemy dep & support airflow 2.3 Oct 25, 2022

hsheth2 force-pushed the loosen-airflow branch from b89b13c to 4e792f4 Compare November 2, 2022 04:34

hsheth2 force-pushed the loosen-airflow branch from 4e792f4 to 7797a99 Compare November 2, 2022 04:46

hsheth2 added 5 commits November 1, 2022 23:59

fix compat

e81284f

save file

451e153

fix tests

080fbb0

move datahub plugin core into main codebase

e57a474

fix lint

62ed57e

hsheth2 marked this pull request as ready for review November 3, 2022 06:17

hsheth2 added 7 commits November 2, 2022 23:20

fix plugin

184e846

fix plugin location

a483f2f

cleanup dep setup

7a40495

test with airflow 2.4

f4c76bb

Merge branch 'master' into loosen-airflow

6ba3446

remove unused method

44462c6

possibly useless commit

2ee6589

Merge branch 'master' into loosen-airflow

c7edcf2

vincentkoc approved these changes Nov 10, 2022

View reviewed changes

tommy-watts-depop approved these changes Nov 10, 2022

View reviewed changes

Making mypy happy

96e2622

hsheth2 changed the title ~~feat(ingest): loosen sqlalchemy dep & support airflow 2.3~~ feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ Nov 11, 2022

hsheth2 added 3 commits November 11, 2022 12:29

include sinks in airflow plugin

4196e71

Revert "include sinks in airflow plugin"

aa73660

This reverts commit 4196e71.

include hooks in airflow setup.py

6e91c86

treff7es approved these changes Nov 11, 2022

View reviewed changes

hsheth2 merged commit 3e907ab into datahub-project:master Nov 11, 2022

hsheth2 deleted the loosen-airflow branch November 11, 2022 20:04

hsheth2 added a commit to hsheth2/datahub that referenced this pull request Nov 12, 2022

fix(ingest/snowflake): loosen snowflake connector version requirement

76bc91f

Related to datahub-project#6204.

hsheth2 mentioned this pull request Nov 12, 2022

fix(ingest/snowflake): loosen snowflake connector version requirement #6418

Merged

5 tasks

treff7es pushed a commit that referenced this pull request Nov 14, 2022

fix(ingest): snowflake - loosen snowflake connector version requireme…

fd518cd

…nt (#6418) Related to #6204.

cccs-Dustin pushed a commit to CybercentreCanada/datahub that referenced this pull request Feb 1, 2023

feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ (datahub-p…

d833b56

…roject#6204) Co-authored-by: Tamas Nemeth <[email protected]>

cccs-Dustin pushed a commit to CybercentreCanada/datahub that referenced this pull request Feb 1, 2023

fix(ingest): snowflake - loosen snowflake connector version requireme…

cf46133

…nt (datahub-project#6418) Related to datahub-project#6204.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ #6204

feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ #6204

hsheth2 commented Oct 14, 2022 •

edited

Loading

github-actions bot commented Oct 14, 2022 •

edited

Loading

github-actions bot commented Oct 14, 2022 •

edited

Loading

vincentkoc left a comment

tommy-watts-depop commented Oct 25, 2022

hsheth2 commented Oct 27, 2022 •

edited

Loading

Vimpy commented Nov 1, 2022

KulykDmytro commented Nov 8, 2022

Vimpy commented Nov 9, 2022

vincentkoc left a comment

vincentkoc commented Nov 10, 2022

vincentkoc commented Nov 10, 2022

keerthiis commented Nov 11, 2022 •

edited

Loading

treff7es left a comment

hsheth2 commented Nov 11, 2022

feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ #6204

feat(ingest): loosen sqlalchemy dep & support airflow 2.3+ #6204

Conversation

hsheth2 commented Oct 14, 2022 • edited Loading

Checklist

github-actions bot commented Oct 14, 2022 • edited Loading

Unit Test Results (metadata ingestion)

github-actions bot commented Oct 14, 2022 • edited Loading

Unit Test Results (build & test)

vincentkoc left a comment

Choose a reason for hiding this comment

tommy-watts-depop commented Oct 25, 2022

hsheth2 commented Oct 27, 2022 • edited Loading

Vimpy commented Nov 1, 2022

KulykDmytro commented Nov 8, 2022

Vimpy commented Nov 9, 2022

vincentkoc left a comment

Choose a reason for hiding this comment

vincentkoc commented Nov 10, 2022

vincentkoc commented Nov 10, 2022

keerthiis commented Nov 11, 2022 • edited Loading

treff7es left a comment

Choose a reason for hiding this comment

hsheth2 commented Nov 11, 2022

hsheth2 commented Oct 14, 2022 •

edited

Loading

github-actions bot commented Oct 14, 2022 •

edited

Loading

github-actions bot commented Oct 14, 2022 •

edited

Loading

hsheth2 commented Oct 27, 2022 •

edited

Loading

keerthiis commented Nov 11, 2022 •

edited

Loading