Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(airflow): remove verbose log from airflow plugin #6516

Merged

Conversation

bskim45
Copy link
Contributor

@bskim45 bskim45 commented Nov 22, 2022

This PR removes the verbose log from the airflow plugin.
These logs are pretty verbose since task_policy are executed for all tasks in dags whenever the airflow CLI command is executed.

image

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Nov 22, 2022
@github-actions
Copy link

github-actions bot commented Nov 22, 2022

Unit Test Results (metadata ingestion)

       8 files  ±0         8 suites  ±0   1h 1m 57s ⏱️ - 1m 46s
   765 tests +1     763 ✔️ +3  2 💤 ±0  0  - 2 
1 532 runs  +2  1 527 ✔️ +4  5 💤 ±0  0  - 2 

Results for commit 4fdd7ac. ± Comparison against base commit d424edd.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Nov 22, 2022

Unit Test Results (build & test)

621 tests  ±0   617 ✔️ ±0   15m 45s ⏱️ -10s
157 suites ±0       4 💤 ±0 
157 files   ±0       0 ±0 

Results for commit 6227813. ± Comparison against base commit d424edd.

♻️ This comment has been updated with latest results.

Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

had some minor questions, but thanks for taking this on

@@ -275,7 +275,6 @@ def custom_on_success_callback(context):


def task_policy(task: BaseOperator) -> None:
print(f"Setting task policy for Dag: {task.dag_id} Task: {task.task_id}")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can change it to logger.debug?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -302,8 +301,6 @@ def _patch_policy(settings):


def _patch_datahub_policy():
print("Patching datahub policy")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imo this one should still be logger.info - I believe this one only gets printed once (as opposed to per dag per task), so a higher log level is ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bskim45 bskim45 requested a review from hsheth2 November 26, 2022 12:57
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, thanks!

@hsheth2 hsheth2 merged commit 6dd6bfc into datahub-project:master Nov 29, 2022
cccs-Dustin pushed a commit to CybercentreCanada/datahub that referenced this pull request Feb 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants