-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add column comments to Snowflake views #53
Add column comments to Snowflake views #53
Conversation
* Update create_view_as macro to include column comments * Add test for column-level view comments * Add testing for column comments with non-lowercase column names
Closing and re-opening to trigger adapter integration tests, now that I've added Edit: There are some unrelated test failures, related to recent changes to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work so far on this @spencer-taylor-workrise! Thanks for proving out the approach. A few comments from me, thinking through best ways to keep the logic organized
|
||
{% set query_columns = get_columns_in_query(sql + ' limit 0') %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Let's avoid running this query if not
config.persist_column_docs()
, most likely by nesting it within the existing conditional below - The
get_columns_in_query
macro already handles addingwhere false limit 0
+ wrapping in a subquery
{% set query_columns = get_columns_in_query(sql + ' limit 0') %} | |
{% set query_columns = get_columns_in_query(sql) %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in 66312f7.
{% for column_name in query_columns %} | ||
{% if (column_name|upper in model_columns) or (column_name in model_columns) %} | ||
{{ adapter.quote(column_name) if model_columns[column_name]['quote'] else column_name }} COMMENT '{{ model_columns[column_name]['description']}}' | ||
{% else %} | ||
{{column_name}} | ||
{% endif %} | ||
{{ ", " if not loop.last else "" }} | ||
{% endfor %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work sketching this out! I think we should have two goals:
- avoid duplicating logic between here +
snowflake__alter_column_comment
down below - keep the
create_view_as
macro clean + legible
I've got some pseudo-code below with ideas of how to do it. Feel free to take/leave/adapt as you see fit.
First, we can wrap the common bits of comment generation in a new macro:
{% macro get_column_comment_sql(column_name, column_dict) %}
{{ adapter.quote(column_name) if column_dict[column_name]['quote'] else column_name }} COMMENT $${{ column_dict[column_name]['description'] | replace('$', '[$]') }}$$
{% endmacro %}
Then, create_view_as
can call a separate macro if we're persisting column-level docs:
{% if config.persist_column_docs() -%}
{% set model_columns = model.columns %}
{% set query_columns = get_columns_in_query(sql) %}
{{ get_persist_docs_column_list(model_columns, query_columns) }}
{% endif %}
That macro would look like:
{% macro get_persist_docs_column_list(model_columns, query_columns) %}
{% for column_name in query_columns %}
{% if (column_name|upper in model_columns) or (column_name in model_columns) %}
{{ get_column_comment_sql(column_name, model_columns) }}
{% else %}
{{ column_name }}
{% endif %}
{{ ", " if not loop.last else "" }}
{% endfor %}
{% endmacro %}
And then the other macro down below:
{% macro snowflake__alter_column_comment(relation, column_dict) -%}
{% set existing_columns = adapter.get_columns_in_relation(relation) | map(attribute="name") | list %}
alter {{ relation.type }} {{ relation }} alter
{% for column_name in column_dict if (column_name in existing_columns) or (column_name|upper in existing_columns) %}
{{ get_column_comment_sql(column_name, column_dict) }} {{ ',' if not loop.last else ';' }}
{% endfor %}
{% endmacro %}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in 66312f7.
@@ -42,11 +42,27 @@ | |||
{%- set secure = config.get('secure', default=false) -%} | |||
{%- set copy_grants = config.get('copy_grants', default=false) -%} | |||
{%- set sql_header = config.get('sql_header', none) -%} | |||
{% set existing_columns = adapter.get_columns_in_relation(relation) | map(attribute="name") | list %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to get existing_columns
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope, I think this is left over from a previous version I was testing out. Removing this shortly.
Hey @jtcohen6 ! Thanks for the comments -- completely agree with the logic there. These have all been resolved in the most recent commit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@spencer-taylor-workrise Nice work, on a quick turnaround! This looks good to me :)
* Slack alerts for failed nightly tests (#50) * Fix: Support individual query tag configuration for Seeds and Snapshots (#48) * Add Query Tag for Seed * Add Query Tag for Snapshots * Add Changelog for #48 * Add Snapshot with Query Tag test * Add Seed with Query Tag test * Update connections.py (#49) * Update connections.py Resolves issues when the Snowflake OCSP server is not reachable and the dbt user would like to run commands without checking the OCSP server. OCSP failures/inaccessibilitiy are possible due to network routing issues or when corporate network security has exposed Snowflake (especially in privatelink) but not the OCSP server. * Update CHANGELOG.md Adding changelog information * Update CHANGELOG.md Co-authored-by: Jeremy Cohen <[email protected]> * Update changelog and unit test Updated changelog and unit tests as requested. * Fix typo Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> * Add column comments to Snowflake views (#53) * Add column comments to Snowflake views * Update create_view_as macro to include column comments * Add test for column-level view comments * Add testing for column comments with non-lowercase column names * Organizing adapter macros for column persistence * Refactor query tag tests (#57) * Refactor query tag tests * Try this * Try storing failures, too * Keep trying * Try it this way * Bumping version to 1.0.0rc2 (#56) * Bumping version to 1.0.0rc2 * Update changelog Co-authored-by: Github Build Bot <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Anton Huck <[email protected]> Co-authored-by: R. Joshua Huntley <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Jeremy Cohen <[email protected]> Co-authored-by: Spencer Taylor <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Github Build Bot <[email protected]>
resolves #17
Description
Adds functionality to persist column descriptions to Snowflake views upon creation.
Expands column name testing (upper/lowercase) for the column-level description integration tests.
Checklist
CHANGELOG.md
and added information about my change to the "dbt-snowflake next" section.