Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add column comments to Snowflake views #53

Merged
merged 4 commits into from
Nov 19, 2021

Conversation

spencer-taylor-workrise
Copy link
Contributor

resolves #17

Description

Adds functionality to persist column descriptions to Snowflake views upon creation.
Expands column name testing (upper/lowercase) for the column-level description integration tests.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt-snowflake next" section.

* Update create_view_as macro to include column comments
* Add test for column-level view comments
* Add testing for column comments with non-lowercase column names
@jtcohen6
Copy link
Contributor

jtcohen6 commented Nov 17, 2021

Closing and re-opening to trigger adapter integration tests, now that I've added ok to test label

Edit: There are some unrelated test failures, related to recent changes to dbt-core. We'll work to get those sorted shortly

@jtcohen6 jtcohen6 closed this Nov 17, 2021
@jtcohen6 jtcohen6 reopened this Nov 17, 2021
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work so far on this @spencer-taylor-workrise! Thanks for proving out the approach. A few comments from me, thinking through best ways to keep the logic organized


{% set query_columns = get_columns_in_query(sql + ' limit 0') %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Let's avoid running this query if not config.persist_column_docs(), most likely by nesting it within the existing conditional below
  2. The get_columns_in_query macro already handles adding where false limit 0 + wrapping in a subquery
Suggested change
{% set query_columns = get_columns_in_query(sql + ' limit 0') %}
{% set query_columns = get_columns_in_query(sql) %}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 66312f7.

Comment on lines 55 to 62
{% for column_name in query_columns %}
{% if (column_name|upper in model_columns) or (column_name in model_columns) %}
{{ adapter.quote(column_name) if model_columns[column_name]['quote'] else column_name }} COMMENT '{{ model_columns[column_name]['description']}}'
{% else %}
{{column_name}}
{% endif %}
{{ ", " if not loop.last else "" }}
{% endfor %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work sketching this out! I think we should have two goals:

  • avoid duplicating logic between here + snowflake__alter_column_comment down below
  • keep the create_view_as macro clean + legible

I've got some pseudo-code below with ideas of how to do it. Feel free to take/leave/adapt as you see fit.

First, we can wrap the common bits of comment generation in a new macro:

{% macro get_column_comment_sql(column_name, column_dict) %}
    {{ adapter.quote(column_name) if column_dict[column_name]['quote'] else column_name }} COMMENT $${{ column_dict[column_name]['description'] | replace('$', '[$]') }}$$
{% endmacro %}

Then, create_view_as can call a separate macro if we're persisting column-level docs:

{% if config.persist_column_docs() -%}
    {% set model_columns = model.columns %}
    {% set query_columns = get_columns_in_query(sql) %}
    {{ get_persist_docs_column_list(model_columns, query_columns) }}
{% endif %} 

That macro would look like:

{% macro get_persist_docs_column_list(model_columns, query_columns) %}
    {% for column_name in query_columns %}
        {% if (column_name|upper in model_columns) or (column_name in model_columns) %}
          {{ get_column_comment_sql(column_name, model_columns) }}  
        {% else %}
          {{ column_name }}
        {% endif %}
        {{ ", " if not loop.last else "" }}
      {% endfor %}
{% endmacro %}

And then the other macro down below:

{% macro snowflake__alter_column_comment(relation, column_dict) -%}
    {% set existing_columns = adapter.get_columns_in_relation(relation) | map(attribute="name") | list %}
    alter {{ relation.type }} {{ relation }} alter
    {% for column_name in column_dict if (column_name in existing_columns) or (column_name|upper in existing_columns) %}
        {{ get_column_comment_sql(column_name, column_dict) }} {{ ',' if not loop.last else ';' }}
    {% endfor %}
{% endmacro %}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 66312f7.

@@ -42,11 +42,27 @@
{%- set secure = config.get('secure', default=false) -%}
{%- set copy_grants = config.get('copy_grants', default=false) -%}
{%- set sql_header = config.get('sql_header', none) -%}
{% set existing_columns = adapter.get_columns_in_relation(relation) | map(attribute="name") | list %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to get existing_columns?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, I think this is left over from a previous version I was testing out. Removing this shortly.

@spencer-taylor-workrise
Copy link
Contributor Author

Hey @jtcohen6 ! Thanks for the comments -- completely agree with the logic there. These have all been resolved in the most recent commit.

Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spencer-taylor-workrise Nice work, on a quick turnaround! This looks good to me :)

@jtcohen6 jtcohen6 merged commit 55f478b into dbt-labs:main Nov 19, 2021
leahwicz added a commit that referenced this pull request Dec 3, 2021
* Slack alerts for failed nightly tests (#50)

* Fix: Support individual query tag configuration for Seeds and Snapshots (#48)

* Add Query Tag for Seed

* Add Query Tag for Snapshots

* Add Changelog for #48

* Add Snapshot with Query Tag test

* Add Seed with Query Tag test

* Update connections.py (#49)

* Update connections.py

Resolves issues when the Snowflake OCSP server is not reachable and the dbt user would like to run commands without checking the OCSP server. OCSP failures/inaccessibilitiy are possible due to network routing issues or when corporate network security has exposed Snowflake (especially in privatelink) but not the OCSP server.

* Update CHANGELOG.md

Adding changelog information

* Update CHANGELOG.md

Co-authored-by: Jeremy Cohen <[email protected]>

* Update changelog and unit test

Updated changelog and unit tests as requested.

* Fix typo

Co-authored-by: Jeremy Cohen <[email protected]>
Co-authored-by: Jeremy Cohen <[email protected]>

* Add column comments to Snowflake views (#53)

* Add column comments to Snowflake views
* Update create_view_as macro to include column comments
* Add test for column-level view comments
* Add testing for column comments with non-lowercase column names

* Organizing adapter macros for column persistence

* Refactor query tag tests (#57)

* Refactor query tag tests

* Try this

* Try storing failures, too

* Keep trying

* Try it this way

* Bumping version to 1.0.0rc2 (#56)

* Bumping version to 1.0.0rc2

* Update changelog

Co-authored-by: Github Build Bot <[email protected]>
Co-authored-by: Jeremy Cohen <[email protected]>

Co-authored-by: Anton Huck <[email protected]>
Co-authored-by: R. Joshua Huntley <[email protected]>
Co-authored-by: Jeremy Cohen <[email protected]>
Co-authored-by: Jeremy Cohen <[email protected]>
Co-authored-by: Spencer Taylor <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Snowflake Persist Docs (Materialization == Views, column level descriptions)
2 participants