Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

amundsendatabuilder -> hive_sample_dag is broken since HiveWatermark class is removed now #550

Closed
pPanda-beta opened this issue Jul 12, 2020 · 4 comments
Closed

amundsendatabuilder -> hive_sample_dag is broken since HiveWatermark class is removed now #550

pPanda-beta opened this issue Jul 12, 2020 · 4 comments
Labels
status:completed Issue is completed and on master type:bug An unexpected problem or unintended behavior

Comments

@pPanda-beta
Copy link
Contributor

This sample sql is not gonna work anymore

https://github.com/lyft/amundsendatabuilder/blob/21a763add3c00c34b4f4c2d9809f59e50fb264c8/example/dags/hive_sample_dag.py#L65-L79

Expected Behavior

ResultRows of the sql are mapped to Watermark models

Current Behavior

It breaks since compulsory constructor parameter 'database' of Watermark class is not provided

Possible Solution

Add a constant value column to sql query

SELECT from_unixtime(A0.CREATE_TIME) as create_time,
       'hive'                        as `database`, -- <<<<<<<<<<<<<<<<<< Solution
       C0.NAME                       as `schema`,
       B0.TBL_NAME                   as table_name,

Steps to Reproduce

Just run the dag with your metastore db

Screenshots (if appropriate)

Context

Your Environment

  • Amundsen version used:
    dockerimages :
    neo4j:3.3.0
    elasticsearch:6.7.0
    amundsendev/amundsen-search:2.4.0
    amundsendev/amundsen-metadata:2.5.4
    amundsendev/amundsen-frontend:2.1.1
    Pypi:
    amundsen-databuilder==2.6.4

  • Data warehouse stores: hive metastore 2.3.7 on mysql by cloudsql

  • Deployment (k8s or native): docker-compose

  • Link to your fork or repository:
@feng-tao
Copy link
Member

@pPanda-beta could you create a pr to fix the issue?
I think we just need to add 'hive' as database, to the SQL (right before https://github.com/lyft/amundsendatabuilder/blob/21a763add3c00c34b4f4c2d9809f59e50fb264c8/example/dags/hive_sample_dag.py#L67 ).

@feng-tao feng-tao added good first issue Good for newcomers help wanted Extra attention is needed Project: Databuilder status:in_progress Issue that is being worked on right now type:bug An unexpected problem or unintended behavior labels Jul 14, 2020
@pPanda-beta
Copy link
Contributor Author

@feng-tao
Thats just one line below where I used 😆 ,
Sure no problem, I'll raise the PR.

@feng-tao
Copy link
Member

thanks

pPanda-beta added a commit to pPanda-beta/amundsendatabuilder that referenced this issue Jul 19, 2020
@pPanda-beta
Fix sql for missing columns and mysql based dialects (#550)
9faf470 
This solves amundsen-io/amundsen#550
@pPanda-beta pPanda-beta mentioned this issue Jul 19, 2020
Merged
5 tasks
@pPanda-beta
Copy link
Contributor Author

@feng-tao -> Raised the PR : amundsen-io/amundsendatabuilder#305

feng-tao pushed a commit to amundsen-io/amundsendatabuilder that referenced this issue Jul 20, 2020
@pPanda-beta
fix: Fix sql for missing columns and mysql based dialects (#550) (#305)
4b7b147 
This solves amundsen-io/amundsen#550
dorianj pushed a commit to dorianj/amundsen that referenced this issue Apr 25, 2021
@pPanda-beta
fix: Fix sql for missing columns and mysql based dialects (amundsen-i…
1c629c5 
…o#550) (amundsen-io#305)

This solves amundsen-io#550
dorianj pushed a commit to dorianj/amundsen that referenced this issue Apr 25, 2021
@Golodhros
feat: Announcements in Homepage (amundsen-io#591)
9e35c7e 
* feat: AnnouncementsList component (amundsen-io#540)

* Adds fake endpoint return for development

* Basic Announcements list

* Basic unstyled Announcements list

* Restoring proper announcements endpoint code

* Linting issues

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Announcements container and saga, api and reducer modifications (amundsen-io#541)

* Basic container

* Adding status code to announcement response

* Updating Announcements reducer, sagas and api to support loading and error states

* Linting details

* Completing the global state fixture

* Basic tests for connection

Signed-off-by: Marcos Iglesias <[email protected]>

* Basic card with loading shimmer (amundsen-io#546)

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Card Styling for announcements (amundsen-io#550)

* Shimmering card styles

* Basic card typography styling

* Add links and link styles to cards

* Adjusting card copy per Knowl specs

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Wiring announcements block on Homepage (amundsen-io#551)

* Adds announcemetns to homepage, integrates basic card

* Spacing and details

* Adjusting loading state

* Adds card and see more links logging

* Updates layout size; focus detail

* Variables on List component styles

* Cleaning fake response

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Adds config for announcements (amundsen-io#562)

* Moves config tests, adds config for announcements and test

Signed-off-by: Marcos Iglesias <[email protected]>

* Wiring announcements feature to the config option

Signed-off-by: Marcos Iglesias <[email protected]>

* Updating configuration docs

Signed-off-by: Marcos Iglesias <[email protected]>

* Extracting navLinks logic into the config utils

Signed-off-by: Marcos Iglesias <[email protected]>

* Adds LPL typography to announcements; some layout fine-tuning (amundsen-io#581)

Signed-off-by: Marcos Iglesias Valle <[email protected]>
Signed-off-by: Marcos Iglesias <[email protected]>

* Fixing merge conflicts

Signed-off-by: Marcos Iglesias <[email protected]>
feng-tao pushed a commit that referenced this issue May 7, 2021
@pPanda-beta
fix: Fix sql for missing columns and mysql based dialects (#550) (#305)
a917774 
This solves #550
feng-tao pushed a commit that referenced this issue May 7, 2021
@Golodhros
feat: Announcements in Homepage (#591)
f3216ce 
* feat: AnnouncementsList component (#540)

* Adds fake endpoint return for development

* Basic Announcements list

* Basic unstyled Announcements list

* Restoring proper announcements endpoint code

* Linting issues

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Announcements container and saga, api and reducer modifications (#541)

* Basic container

* Adding status code to announcement response

* Updating Announcements reducer, sagas and api to support loading and error states

* Linting details

* Completing the global state fixture

* Basic tests for connection

Signed-off-by: Marcos Iglesias <[email protected]>

* Basic card with loading shimmer (#546)

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Card Styling for announcements (#550)

* Shimmering card styles

* Basic card typography styling

* Add links and link styles to cards

* Adjusting card copy per Knowl specs

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Wiring announcements block on Homepage (#551)

* Adds announcemetns to homepage, integrates basic card

* Spacing and details

* Adjusting loading state

* Adds card and see more links logging

* Updates layout size; focus detail

* Variables on List component styles

* Cleaning fake response

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Adds config for announcements (#562)

* Moves config tests, adds config for announcements and test

Signed-off-by: Marcos Iglesias <[email protected]>

* Wiring announcements feature to the config option

Signed-off-by: Marcos Iglesias <[email protected]>

* Updating configuration docs

Signed-off-by: Marcos Iglesias <[email protected]>

* Extracting navLinks logic into the config utils

Signed-off-by: Marcos Iglesias <[email protected]>

* Adds LPL typography to announcements; some layout fine-tuning (#581)

Signed-off-by: Marcos Iglesias Valle <[email protected]>
Signed-off-by: Marcos Iglesias <[email protected]>

* Fixing merge conflicts

Signed-off-by: Marcos Iglesias <[email protected]>
hansadriaans pushed a commit to DataChefHQ/amundsen that referenced this issue Jun 30, 2022
@pPanda-beta
fix: Fix sql for missing columns and mysql based dialects (amundsen-i…
784056d 
…o#550) (amundsen-io#305)

This solves amundsen-io#550
hansadriaans pushed a commit to DataChefHQ/amundsen that referenced this issue Jun 30, 2022
@Golodhros
feat: Announcements in Homepage (amundsen-io#591)
f3f3761 
* feat: AnnouncementsList component (amundsen-io#540)

* Adds fake endpoint return for development

* Basic Announcements list

* Basic unstyled Announcements list

* Restoring proper announcements endpoint code

* Linting issues

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Announcements container and saga, api and reducer modifications (amundsen-io#541)

* Basic container

* Adding status code to announcement response

* Updating Announcements reducer, sagas and api to support loading and error states

* Linting details

* Completing the global state fixture

* Basic tests for connection

Signed-off-by: Marcos Iglesias <[email protected]>

* Basic card with loading shimmer (amundsen-io#546)

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Card Styling for announcements (amundsen-io#550)

* Shimmering card styles

* Basic card typography styling

* Add links and link styles to cards

* Adjusting card copy per Knowl specs

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Wiring announcements block on Homepage (amundsen-io#551)

* Adds announcemetns to homepage, integrates basic card

* Spacing and details

* Adjusting loading state

* Adds card and see more links logging

* Updates layout size; focus detail

* Variables on List component styles

* Cleaning fake response

Signed-off-by: Marcos Iglesias <[email protected]>

* feat: Adds config for announcements (amundsen-io#562)

* Moves config tests, adds config for announcements and test

Signed-off-by: Marcos Iglesias <[email protected]>

* Wiring announcements feature to the config option

Signed-off-by: Marcos Iglesias <[email protected]>

* Updating configuration docs

Signed-off-by: Marcos Iglesias <[email protected]>

* Extracting navLinks logic into the config utils

Signed-off-by: Marcos Iglesias <[email protected]>

* Adds LPL typography to announcements; some layout fine-tuning (amundsen-io#581)

Signed-off-by: Marcos Iglesias Valle <[email protected]>
Signed-off-by: Marcos Iglesias <[email protected]>

* Fixing merge conflicts

Signed-off-by: Marcos Iglesias <[email protected]>
@Golodhros Golodhros added status:completed Issue is completed and on master and removed help wanted Extra attention is needed good first issue Good for newcomers status:in_progress Issue that is being worked on right now labels Dec 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status:completed Issue is completed and on master type:bug An unexpected problem or unintended behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Golodhros @feng-tao @pPanda-beta