Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Sample Scripts and Data #199

Merged
merged 5 commits into from
Feb 25, 2020

Conversation

jacobhjkim
Copy link
Contributor

@jacobhjkim jacobhjkim commented Feb 24, 2020

Summary of Changes

Currently, sample_data_loader.py is bugged. Therefore, I made some fixes.

This PR mostly changed schema_name -> schema and name -> full_name.

If you run sample_data_loader.py on the latest branch without making any changes, it will return this error.

Traceback (most recent call last):
  File "example/scripts/sample_data_loader.py", line 643, in <module>
    load_table_data_from_csv('sample_table_programmatic_source.csv', 'programmatic')
  File "example/scripts/sample_data_loader.py", line 89, in load_table_data_from_csv
    i['description_source']) for i in dr]
  File "example/scripts/sample_data_loader.py", line 89, in <listcomp>
    i['description_source']) for i in dr]
KeyError: 'schema'

Documentation / Unit Test

No need for either since this only changes example scripts.

Checklist

Make sure you have checked all steps below to ensure a timely review.

  • PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
  • PR includes a summary of changes.
  • PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
  • PR passes make test

@jacobhjkim jacobhjkim changed the title Fix sample script schema Fix Sample Scripts and Data Feb 24, 2020
@jacobhjkim
Copy link
Contributor Author

jacobhjkim commented Feb 24, 2020

However, there is still an issue in sample_data_loader.py and I am trying to fix it.
This one is also fixed.

Traceback (most recent call last):
  File "example/scripts/sample_data_loader.py", line 807, in <module>
    job_es_user.launch()
  File "/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.0.0-py3.7.egg/databuilder/job/job.py", line 78, in launch
  File "/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.0.0-py3.7.egg/databuilder/job/job.py", line 68, in launch
  File "/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.0.0-py3.7.egg/databuilder/task/task.py", line 58, in run
  File "/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.0.0-py3.7.egg/databuilder/extractor/neo4j_search_data_extractor.py", line 75, in extract
  File "/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.0.0-py3.7.egg/databuilder/extractor/neo4j_extractor.py", line 101, in extract
  File "/usr/local/lib/python3.7/site-packages/amundsen_databuilder-2.0.0-py3.7.egg/databuilder/extractor/neo4j_extractor.py", line 86, in _get_extract_iter
TypeError: __init__() got an unexpected keyword argument 'name'

@codecov-io
Copy link

codecov-io commented Feb 24, 2020

Codecov Report

Merging #199 into master will decrease coverage by 1.44%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #199      +/-   ##
==========================================
- Coverage   81.62%   80.18%   -1.45%     
==========================================
  Files          73       79       +6     
  Lines        3511     3669     +158     
  Branches      383      391       +8     
==========================================
+ Hits         2866     2942      +76     
- Misses        537      616      +79     
- Partials      108      111       +3
Impacted Files Coverage Δ
databuilder/rest_api/rest_api_query.py 82.85% <0%> (-2.33%) ⬇️
...er/extractor/dashboard/mode_dashboard_extractor.py
...atabuilder/models/dashboard/dashboard_execution.py 71.05% <0%> (ø)
databuilder/transformer/dict_to_model.py 94.11% <0%> (ø)
...tabuilder/transformer/timestamp_string_to_epoch.py 87.5% <0%> (ø)
...e_analytics/mode_dashboard_executions_extractor.py 0% <0%> (ø)
...r/dashboard/mode_analytics/mode_dashboard_utils.py 0% <0%> (ø)
...shboard/mode_analytics/mode_dashboard_extractor.py 0% <0%> (ø)
...shboard/mode_analytics/mode_dashboard_constants.py 0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dac8110...dac961a. Read the comment docs.

@samshuster
Copy link
Contributor

Just noticed this as well. Heads up @jinhyukchang @feng-tao

@feng-tao
Copy link
Member

@TheJacobKim lgtm, could you fix the CLA check(normally due to not config github user/email in your local env)? Thanks.

Copy link
Contributor

@friendtocephalopods friendtocephalopods left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix!

feng-tao
feng-tao previously approved these changes Feb 24, 2020
@jacobhjkim
Copy link
Contributor Author

Thanks @feng-tao, the CLA check issue was with my local git config. Is there a more elegant way to trigger the CLA check other than pushing a commit?

@feng-tao
Copy link
Member

thanks @TheJacobKim ! I think that's the easier approach.

@feng-tao feng-tao merged commit 6d5ffa4 into amundsen-io:master Feb 25, 2020
ccarterlandis pushed a commit to Gusto/amundsendatabuilder_contrib that referenced this pull request Aug 7, 2020
* Patch to fix backward incompatible issue on table metadata (amundsen-io#195)

* Backward compatible patch

* Increment version

* Update

* escaping backslashes in neo4j publisher (amundsen-io#191)

* escaping backslashes in neo4j publisher

* removing an extra backslash

* Bumping versions to 1.5.3

* Mode Dashboard extractor with Generic REST API Query (amundsen-io#194)

* Initial check in on REST API Query

* Working version

* docstring

* Update

* Update

* Make unit test happy

* Update docstring

* Update

* Update

* Update

* Adding unit tests

* Updated README.md

* jsonpath_rw to extra_requires

* update models for 2.0.0 (amundsen-io#188)

* update models for 2.0.0

schema_name -> schema in table model
update search model to use schema, last_updated_timestamp

* update name -> full_name

* Mode Dashboard execution timestamp and state (amundsen-io#197)

* Initial commit

* Update

* Update

* Update

* flake8

* Update docstring

* Update

* Update

* Add mode_dashboard_constants and mode_dashboard_utils

* Add more to Util

* Added back model class to ModeDashboardExtractor

* flake8

* Address PR comments

* Fix Sample Scripts and Data (amundsen-io#199)

* Change schema_name -> schema in sample_table_programmatic_source.csv

* Update schema_name -> schema in sample_data_loader

* Change name -> full_name in  sample_user.csv

* Change name -> full_name in sample_data_loader.py

* Retrigger CLA Check

* Fix broken HiveTableMetadataExtractor. (amundsen-io#201)

* Fix.

* Update corresponding tests.

* Replace schema_name with  and fix presto as well.

* Replace unit test with schema, since we renamed schema_name back to .

* Update setup.py

Co-authored-by: Tao Feng <[email protected]>

* Follow up patches on v2 field name changes (amundsen-io#202)

* Mode dashboard owner (amundsen-io#200)

* Mode dashboard owner

* Not having dashboard_owner to create User node

* Update

* Support unicode in file_system_neo4j_csv_loader (amundsen-io#203)

* Support unicode in file_system_neo4j_csv_loader

* increment version

* Update

* Update

* Let Python3 use csv not unicodecsv

* Add badges to Neo4jExtractor and elastic search  (amundsen-io#204)

* Add badges to Neo4jSearchExtractor

* update publisher to have badges

* update elastic search document

* fix typo

* update name

* filter tags by type

* typo

* do not filter tags because then i can't get badges on staging :|

* update tests

* fix tests

* use amunsen_common for elastic search index

* revert commit using amundsencommon

* add comment

* make backwards compatible

* remove badges from tags

* Mode Dashboard last successful execution (amundsen-io#205)

* Mode Dashboard last successful execution

* Increment version

* Remove py27 Antlar presto sql usage extractor (amundsen-io#208)

* Remove py27 Antlar presto sql usage extractor

* remove more

* remove pytest macro

* Removing Sample Csv Data Loader and Cleaning up Sample Data Loader (amundsen-io#206)

* refactored TableColumnCsvExtractor to the csv_extractors file.
removing sample_csv_data_loader.py
simplified sample_data_loader.py
Finally, modified sample data so that popular table shows by default again.

* Update sample_data_loader.py

Co-authored-by: Tao Feng <[email protected]>

* Added created timestamp and last modified timestamp on Dashboard (amundsen-io#207)

* Added created timestamp and last modified timestamp on Dashboard

* Remove owner and reload time from dashboard_metadata

* Move dashboard_metadata under dashboard package

* Update

* Update

* Added dashboard group url and dashboard url

* Added cluster node in Dashboard

* Optimize Neo4j Cypher query on Neo4jSearchDataExtractor (amundsen-io#213)

* Optimize Neo4j Cypher query on Neo4jSearchDataExtractor

* flake8

* Add Dashboard Table model (amundsen-io#210)

* Add Dashboard Table model

* Update dashboard_table.py

* Update test_dashboard_table.py

* bump version

* Remove antlr4 deps (amundsen-io#214)

* Update dashboard ES doc (amundsen-io#218)

* Remove neo4j dashboard/metric extractor (amundsen-io#219)

* Add default cypher query for user/dashboard entity to search extractor (amundsen-io#220)

* Update the query publish tag to match entity (amundsen-io#221)

* issue-320 documentation of models (amundsen-io#217)

* Start if issue-320 documentatino of models

* adding more documentation

* Adding BigQuery to the readme (amundsen-io#222)

* Adding Bigquery to the readme

* fixing link

* Rename sample_dag.py to hive_sample_dag.py (amundsen-io#223)

To clarify what this sample is about. 
A year ago this folder only had one file :)

* Adding Mode dashboard usage extractor and generic loader (amundsen-io#225)

* Adding Mode dashboard usage extractor and generic loader

* Update

* Increment version

* Add query and chart on Dashboard (amundsen-io#224)

* Add query and chart on Dashboard

* Update

* Increment version

* Update test case

* Add title field in user model (amundsen-io#227)

* Add title field in user model

* fix one more test

* Slight change from title to role name (amundsen-io#228)

* Update python version for databuilder (amundsen-io#229)

* Dashboard usage model (amundsen-io#226)

* Added Dashboard usage model

* Update

* Update

* Update

* Clarify ES publisher warning on first run (amundsen-io#230)

* Add role_name to user ES doc (amundsen-io#232)

* minor mods to get tests to pass

* tweak schema field after testing demo execution

* move Table into simple_sql_parser now that we're not trying to conform to a missing presto parser

* add role_name to sample_data_loader

Co-authored-by: Jin Hyuk Chang <[email protected]>
Co-authored-by: Luke Lowery <[email protected]>
Co-authored-by: friendtocephalopods <[email protected]>
Co-authored-by: Jacob Kim <[email protected]>
Co-authored-by: Robert Yi <[email protected]>
Co-authored-by: Tao Feng <[email protected]>
Co-authored-by: christina stead <[email protected]>
Co-authored-by: Tao Feng <[email protected]>
Co-authored-by: samshuster <[email protected]>
Co-authored-by: jornh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants