issue-297/Adding programmatic_descriptions to table search export #198

samshuster · 2020-02-21T17:41:08Z

Summary of Changes

This adds programmatic_desciptions to the elastic search documents

Tests

Modified existing tests.

Documentation

None

CheckList

Make sure you have checked all steps below to ensure a timely review.

[ x] PR title addresses the issue accurately and concisely. Example: "Updates the version of Flask to v1.0.2"
- In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.
PR includes a summary of changes.
PR adds unit tests, updates existing unit tests, OR documents why no test additions or modifications are needed.
In case of new functionality, my PR adds documentation that describes how to use it.
- All the public functions and the classes in the PR contain docstrings that explain what it does
PR passes make test

codecov-io · 2020-02-21T17:47:04Z

Codecov Report

Merging #198 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #198   +/-   ##
=======================================
  Coverage   74.45%   74.45%           
=======================================
  Files         100      100           
  Lines        4188     4189    +1     
  Branches      387      387           
=======================================
+ Hits         3118     3119    +1     
  Misses        971      971           
  Partials       99       99

Impacted Files	Coverage Δ
...tabuilder/extractor/neo4j_search_data_extractor.py	`89.47% <ø> (ø)`
databuilder/models/table_elasticsearch_document.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update edbb208...e7660f8. Read the comment docs.

samshuster · 2020-02-24T17:19:36Z

marking as WIP as the merge with the most recent branch broke quite a few things

samshuster · 2020-02-25T17:21:15Z

ok this successfully includes upstream changes

samshuster · 2020-03-06T14:13:18Z

here is a change related to amundsen-io/amundsensearchlibrary#83 as part of issue amundsen-io/amundsen#297

@feng-tao

samshuster · 2020-03-17T16:31:43Z

hey @jinhyukchang now that the frontend changes are merged, this one is ready for you to review too. Thanks!

feng-tao · 2020-03-25T05:32:41Z

@ttannis @jinhyukchang could you two help @samshuster to review the change as you two have followed through the discussion? Thanks. I assume this will be the last piece to get this change in?

samshuster · 2020-03-25T14:46:55Z

thanks @feng-tao these 2 prs (this one and search: amundsen-io/amundsensearchlibrary#83) are actually part of a follow up issue here amundsen-io/amundsen#297 to add it programmatic_descriptions to search as that was not included as part of the original issue.

…matic_search

…matic_search # Conflicts: # databuilder/extractor/neo4j_search_data_extractor.py # databuilder/models/table_elasticsearch_document.py # databuilder/publisher/elasticsearch_publisher.py # tests/unit/extractor/test_neo4j_extractor.py # tests/unit/loader/test_file_system_elasticsearch_json_loader.py # tests/unit/models/test_table_elasticsearch_document.py

samshuster · 2020-05-15T19:03:04Z

hey @feng-tao I fixed the merge conflicts on this one and it should be good to go again.

jinhyukchang

Looks good and left a comment.
Could you also update Elasticsearch mapping? https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_constants.py#L10

databuilder/extractor/neo4j_search_data_extractor.py

…#198) Bumps [amundsenfrontendlibrary](https://github.com/lyft/amundsenfrontendlibrary) from `e355bca` to `57f05b1`. - [Release notes](https://github.com/lyft/amundsenfrontendlibrary/releases) - [Commits](amundsen-io/amundsenfrontendlibrary@e355bca...57f05b1) Signed-off-by: dependabot-preview[bot] <[email protected]>

codecov-commenter · 2020-06-01T17:18:34Z

Codecov Report

Merging #198 into master will decrease coverage by 0.33%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #198      +/-   ##
==========================================
- Coverage   74.45%   74.11%   -0.34%     
==========================================
  Files         100      100              
  Lines        4188     4200      +12     
  Branches      387      371      -16     
==========================================
- Hits         3118     3113       -5     
- Misses        971      979       +8     
- Partials       99      108       +9

Impacted Files	Coverage Δ
...tabuilder/extractor/neo4j_search_data_extractor.py	`89.47% <ø> (ø)`
databuilder/publisher/elasticsearch_constants.py	`100.00% <ø> (ø)`
databuilder/models/table_elasticsearch_document.py	`100.00% <100.00%> (ø)`
databuilder/loader/file_system_neo4j_csv_loader.py	`83.33% <0.00%> (-5.96%)`	⬇️
...ilder/transformer/regex_str_replace_transformer.py	`92.85% <0.00%> (-2.98%)`	⬇️
...abuilder/extractor/snowflake_metadata_extractor.py	`93.44% <0.00%> (-1.64%)`	⬇️
...tabuilder/extractor/postgres_metadata_extractor.py	`93.54% <0.00%> (-1.62%)`	⬇️
databuilder/extractor/mssql_metadata_extractor.py	`94.02% <0.00%> (-1.50%)`	⬇️
databuilder/models/dashboard/dashboard_query.py	`84.09% <0.00%> (-1.28%)`	⬇️
databuilder/publisher/neo4j_csv_publisher.py	`84.23% <0.00%> (-0.99%)`	⬇️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update edbb208...9edcd08. Read the comment docs.

jinhyukchang · 2020-06-02T17:47:06Z

Looks good and left a comment.
Could you also update Elasticsearch mapping? https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_constants.py#L10

@samshuster Thanks for the update! Could you update ES mapping as well? (Couldn't find the update) All others look good :)

samshuster · 2020-06-03T15:04:46Z

Looks good and left a comment.
Could you also update Elasticsearch mapping? https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_constants.py#L10

@samshuster Thanks for the update! Could you update ES mapping as well? (Couldn't find the update) All others look good :)

ah whoops missed it! thanks. I added it as type text, analyzer simple.

Should schema description also be added to this file?

jinhyukchang · 2020-06-03T18:05:58Z

Looks good and left a comment.
Could you also update Elasticsearch mapping? https://github.com/lyft/amundsendatabuilder/blob/master/databuilder/publisher/elasticsearch_constants.py#L10

@samshuster Thanks for the update! Could you update ES mapping as well? (Couldn't find the update) All others look good :)

ah whoops missed it! thanks. I added it as type text, analyzer simple.

Should schema description also be added to this file?

Thanks, @samshuster! Yea, IIRC, by default ES auto-detect type but it could make it fail if auto-detect is turned-off.

jinhyukchang

Thanks for the update @samshuster !

samshuster mentioned this pull request Feb 21, 2020

Add Programmatic_Descriptions to Search amundsen-io/amundsen#297

Closed

samshuster force-pushed the issue-297/programmatic_search branch from b20ffcb to 4b0abf6 Compare February 21, 2020 21:19

samshuster changed the title ~~Adding programmatic_descriptions to table search export~~ WIP: Adding programmatic_descriptions to table search export Feb 24, 2020

samshuster force-pushed the issue-297/programmatic_search branch from 4b0abf6 to e879ac0 Compare February 25, 2020 16:44

samshuster changed the title ~~WIP: Adding programmatic_descriptions to table search export~~ Adding programmatic_descriptions to table search export Feb 25, 2020

samshuster force-pushed the issue-297/programmatic_search branch from e879ac0 to 547dfd8 Compare March 6, 2020 13:58

samshuster changed the title ~~Adding programmatic_descriptions to table search export~~ issue-297/Adding programmatic_descriptions to table search export Mar 6, 2020

samshuster added 3 commits March 17, 2020 08:49

Adding programmatic_descriptions to table search export

d13d5d1

fixing tests from merge

2081933

Rebasing from upstream master

495f3be

samshuster force-pushed the issue-297/programmatic_search branch from 490e9a7 to 495f3be Compare March 17, 2020 16:19

feng-tao requested a review from jinhyukchang March 25, 2020 05:30

feng-tao requested review from csteez, ttannis and ajittripathi and removed request for csteez and ajittripathi March 25, 2020 05:32

feng-tao added the keep fresh Disables stalebot from closing an issue label Mar 30, 2020

samshuster added 3 commits May 15, 2020 10:32

Merge remote-tracking branch 'upstream/master' into issue-297/program…

d992ceb

…matic_search

fixing merge

e7660f8

jinhyukchang reviewed May 27, 2020

View reviewed changes

databuilder/extractor/neo4j_search_data_extractor.py Show resolved Hide resolved

fixing the neo4j query to be more optimized

43da034

samshuster requested a review from jinhyukchang June 2, 2020 14:51

adding programmatic_descriptions to the elasticsearch_constants.py

9edcd08

jinhyukchang approved these changes Jun 3, 2020

View reviewed changes

jinhyukchang merged commit 8f18faf into amundsen-io:master Jun 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

issue-297/Adding programmatic_descriptions to table search export #198

issue-297/Adding programmatic_descriptions to table search export #198

samshuster commented Feb 21, 2020

codecov-io commented Feb 21, 2020 •

edited

Loading

samshuster commented Feb 24, 2020

samshuster commented Feb 25, 2020

samshuster commented Mar 6, 2020

samshuster commented Mar 17, 2020

feng-tao commented Mar 25, 2020

samshuster commented Mar 25, 2020

samshuster commented May 15, 2020

jinhyukchang left a comment

codecov-commenter commented Jun 1, 2020 •

edited

Loading

jinhyukchang commented Jun 2, 2020

samshuster commented Jun 3, 2020

jinhyukchang commented Jun 3, 2020

jinhyukchang left a comment

issue-297/Adding programmatic_descriptions to table search export #198

issue-297/Adding programmatic_descriptions to table search export #198

Conversation

samshuster commented Feb 21, 2020

Summary of Changes

Tests

Documentation

CheckList

codecov-io commented Feb 21, 2020 • edited Loading

Codecov Report

samshuster commented Feb 24, 2020

samshuster commented Feb 25, 2020

samshuster commented Mar 6, 2020

samshuster commented Mar 17, 2020

feng-tao commented Mar 25, 2020

samshuster commented Mar 25, 2020

samshuster commented May 15, 2020

jinhyukchang left a comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 1, 2020 • edited Loading

Codecov Report

jinhyukchang commented Jun 2, 2020

samshuster commented Jun 3, 2020

jinhyukchang commented Jun 3, 2020

jinhyukchang left a comment

Choose a reason for hiding this comment

codecov-io commented Feb 21, 2020 •

edited

Loading

codecov-commenter commented Jun 1, 2020 •

edited

Loading