Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dbt Core 1.2 and utils 0.9.0 prep #625

Merged
merged 6 commits into from
Jul 28, 2022
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: 'dbt_utils'
version: '0.1.0'

require-dbt-version: [">=1.0.0", "<2.0.0"]
require-dbt-version: [">=1.2.0-rc1", "<2.0.0"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing this will drop the -rc1 suffix and flip to just >=1.2.0 prior to the final release of dbt_utils 0.9.0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed!

  • dbt-utils 0.9.0-rc1 supports dbt-core 1.2.0-rc1.
  • dbt-utils 0.9.0 will support dbt-core 1.2.0


config-version: 2

Expand Down
3 changes: 1 addition & 2 deletions macros/cross_db_utils/any_value.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro any_value(expression) -%}
{{ return(adapter.dispatch('any_value', 'dbt_utils') (expression)) }}
{% endmacro %}

{% macro default__any_value(expression) -%}
{% do dbt_utils.xdb_deprecation_warning('any_value', model.package_name, model.name) %}
{{ return(adapter.dispatch('any_value', 'dbt') (expression)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/bool_or.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro bool_or(expression) -%}
{{ return(adapter.dispatch('bool_or', 'dbt_utils') (expression)) }}
{% endmacro %}

{% macro default__bool_or(expression) -%}
{% do dbt_utils.xdb_deprecation_warning('bool_or', model.package_name, model.name) %}
{{ return(adapter.dispatch('bool_or', 'dbt') (expression)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/cast_bool_to_text.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro cast_bool_to_text(field) %}
{{ adapter.dispatch('cast_bool_to_text', 'dbt_utils') (field) }}
{% endmacro %}

{% macro default__cast_bool_to_text(field) %}
{% do dbt_utils.xdb_deprecation_warning('cast_bool_to_text', model.package_name, model.name) %}
{{ adapter.dispatch('cast_bool_to_text', 'dbt') (field) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/concat.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro concat(fields) -%}
{{ return(adapter.dispatch('concat', 'dbt_utils')(fields)) }}
{%- endmacro %}

{% macro default__concat(fields) -%}
{% do dbt_utils.xdb_deprecation_warning('concat', model.package_name, model.name) %}
{{ return(adapter.dispatch('concat', 'dbt')(fields)) }}
{%- endmacro %}
9 changes: 6 additions & 3 deletions macros/cross_db_utils/datatypes.sql
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
{# These macros have been moved into dbt-core #}
{# Here for backwards compatibility ONLY #}

{# string ------------------------------------------------- #}

{%- macro type_string() -%}
{{ return(adapter.dispatch('type_string', 'dbt_utils')()) }}
{%- endmacro -%}

{% macro default__type_string() %}
{% do dbt_utils.xdb_deprecation_warning('type_string', model.package_name, model.name) %}
{{ return(adapter.dispatch('type_string', 'dbt')()) }}
{% endmacro %}

Expand All @@ -19,6 +17,7 @@
{%- endmacro -%}

{% macro default__type_timestamp() %}
{% do dbt_utils.xdb_deprecation_warning('type_timestamp', model.package_name, model.name) %}
{{ return(adapter.dispatch('type_timestamp', 'dbt')()) }}
{% endmacro %}

Expand All @@ -30,6 +29,7 @@
{%- endmacro -%}

{% macro default__type_float() %}
{% do dbt_utils.xdb_deprecation_warning('type_float', model.package_name, model.name) %}
{{ return(adapter.dispatch('type_float', 'dbt')()) }}
{% endmacro %}

Expand All @@ -41,6 +41,7 @@
{%- endmacro -%}

{% macro default__type_numeric() %}
{% do dbt_utils.xdb_deprecation_warning('type_numeric', model.package_name, model.name) %}
{{ return(adapter.dispatch('type_numeric', 'dbt')()) }}
{% endmacro %}

Expand All @@ -52,6 +53,7 @@
{%- endmacro -%}

{% macro default__type_bigint() %}
{% do dbt_utils.xdb_deprecation_warning('type_bigint', model.package_name, model.name) %}
{{ return(adapter.dispatch('type_bigint', 'dbt')()) }}
{% endmacro %}

Expand All @@ -63,5 +65,6 @@
{%- endmacro -%}

{% macro default__type_int() %}
{% do dbt_utils.xdb_deprecation_warning('type_int', model.package_name, model.name) %}
{{ return(adapter.dispatch('type_int', 'dbt')()) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/date_trunc.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro date_trunc(datepart, date) -%}
{{ return(adapter.dispatch('date_trunc', 'dbt_utils') (datepart, date)) }}
{%- endmacro %}

{% macro default__date_trunc(datepart, date) -%}
{% do dbt_utils.xdb_deprecation_warning('date_trunc', model.package_name, model.name) %}
{{ return(adapter.dispatch('date_trunc', 'dbt') (datepart, date)) }}
{%- endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/dateadd.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro dateadd(datepart, interval, from_date_or_timestamp) %}
{{ return(adapter.dispatch('dateadd', 'dbt_utils')(datepart, interval, from_date_or_timestamp)) }}
{% endmacro %}

{% macro default__dateadd(datepart, interval, from_date_or_timestamp) %}
{% do dbt_utils.xdb_deprecation_warning('dateadd', model.package_name, model.name) %}
{{ return(adapter.dispatch('dateadd', 'dbt')(datepart, interval, from_date_or_timestamp)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/datediff.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro datediff(first_date, second_date, datepart) %}
{{ return(adapter.dispatch('datediff', 'dbt_utils')(first_date, second_date, datepart)) }}
{% endmacro %}

{% macro default__datediff(first_date, second_date, datepart) %}
{% do dbt_utils.xdb_deprecation_warning('datediff', model.package_name, model.name) %}
{{ return(adapter.dispatch('datediff', 'dbt')(first_date, second_date, datepart)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/escape_single_quotes.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro escape_single_quotes(expression) %}
{{ return(adapter.dispatch('escape_single_quotes', 'dbt_utils') (expression)) }}
{% endmacro %}

{% macro default__escape_single_quotes(expression) %}
{% do dbt_utils.xdb_deprecation_warning('escape_single_quotes', model.package_name, model.name) %}
{{ return(adapter.dispatch('escape_single_quotes', 'dbt') (expression)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/except.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro except() %}
{{ return(adapter.dispatch('except', 'dbt_utils')()) }}
{% endmacro %}

{% macro default__except() %}
{% do dbt_utils.xdb_deprecation_warning('except', model.package_name, model.name) %}
{{ return(adapter.dispatch('except', 'dbt')()) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/hash.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro hash(field) -%}
{{ return(adapter.dispatch('hash', 'dbt_utils') (field)) }}
{%- endmacro %}

{% macro default__hash(field) -%}
{% do dbt_utils.xdb_deprecation_warning('hash', model.package_name, model.name) %}
{{ return(adapter.dispatch('hash', 'dbt') (field)) }}
{%- endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/intersect.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro intersect() %}
{{ return(adapter.dispatch('intersect', 'dbt_utils')()) }}
{% endmacro %}

{% macro default__intersect() %}
{% do dbt_utils.xdb_deprecation_warning('intersect', model.package_name, model.name) %}
{{ return(adapter.dispatch('intersect', 'dbt')()) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/last_day.sql
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
{# This is here for backwards compatibility only #}

/*
This function has been tested with dateparts of month and quarters. Further
testing is required to validate that it will work on other dateparts.
Expand All @@ -10,5 +8,6 @@ testing is required to validate that it will work on other dateparts.
{% endmacro %}

{% macro default__last_day(date, datepart) %}
{% do dbt_utils.xdb_deprecation_warning('last_day', model.package_name, model.name) %}
{{ return(adapter.dispatch('last_day', 'dbt') (date, datepart)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/length.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro length(expression) -%}
{{ return(adapter.dispatch('length', 'dbt_utils') (expression)) }}
{% endmacro %}

{% macro default__length(expression) -%}
{% do dbt_utils.xdb_deprecation_warning('length', model.package_name, model.name) %}
{{ return(adapter.dispatch('length', 'dbt') (expression)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/listagg.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro listagg(measure, delimiter_text="','", order_by_clause=none, limit_num=none) -%}
{{ return(adapter.dispatch('listagg', 'dbt_utils') (measure, delimiter_text, order_by_clause, limit_num)) }}
{%- endmacro %}

{% macro default__listagg(measure, delimiter_text="','", order_by_clause=none, limit_num=none) -%}
{% do dbt_utils.xdb_deprecation_warning('listagg', model.package_name, model.name) %}
{{ return(adapter.dispatch('listagg', 'dbt') (measure, delimiter_text, order_by_clause, limit_num)) }}
{%- endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/literal.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{%- macro string_literal(value) -%}
{{ return(adapter.dispatch('string_literal', 'dbt_utils') (value)) }}
{%- endmacro -%}

{%- macro default__string_literal(value) -%}
{% do dbt_utils.xdb_deprecation_warning('string_literal', model.package_name, model.name) %}
{{ return(adapter.dispatch('string_literal', 'dbt') (value)) }}
{%- endmacro -%}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/position.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro position(substring_text, string_text) -%}
{{ return(adapter.dispatch('position', 'dbt_utils') (substring_text, string_text)) }}
{% endmacro %}

{% macro default__position(substring_text, string_text) -%}
{% do dbt_utils.xdb_deprecation_warning('position', model.package_name, model.name) %}
{{ return(adapter.dispatch('position', 'dbt') (substring_text, string_text)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/replace.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro replace(field, old_chars, new_chars) -%}
{{ return(adapter.dispatch('replace', 'dbt_utils') (field, old_chars, new_chars)) }}
{% endmacro %}

{% macro default__replace(field, old_chars, new_chars) -%}
{% do dbt_utils.xdb_deprecation_warning('replace', model.package_name, model.name) %}
{{ return(adapter.dispatch('replace', 'dbt') (field, old_chars, new_chars)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/right.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro right(string_text, length_expression) -%}
{{ return(adapter.dispatch('right', 'dbt_utils') (string_text, length_expression)) }}
{% endmacro %}

{% macro default__right(string_text, length_expression) -%}
{% do dbt_utils.xdb_deprecation_warning('right', model.package_name, model.name) %}
{{ return(adapter.dispatch('right', 'dbt') (string_text, length_expression)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/safe_cast.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro safe_cast(field, type) %}
{{ return(adapter.dispatch('safe_cast', 'dbt_utils') (field, type)) }}
{% endmacro %}

{% macro default__safe_cast(field, type) %}
{% do dbt_utils.xdb_deprecation_warning('safe_cast', model.package_name, model.name) %}
{{ return(adapter.dispatch('safe_cast', 'dbt') (field, type)) }}
{% endmacro %}
3 changes: 1 addition & 2 deletions macros/cross_db_utils/split_part.sql
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
{# This is here for backwards compatibility only #}

{% macro split_part(string_text, delimiter_text, part_number) %}
{{ return(adapter.dispatch('split_part', 'dbt_utils') (string_text, delimiter_text, part_number)) }}
{% endmacro %}

{% macro default__split_part(string_text, delimiter_text, part_number) %}
{% do dbt_utils.xdb_deprecation_warning('split_part', model.package_name, model.name) %}
{{ return(adapter.dispatch('split_part', 'dbt') (string_text, delimiter_text, part_number)) }}
{% endmacro %}
4 changes: 4 additions & 0 deletions macros/cross_db_utils/xdb_deprecation_warning.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{% macro xdb_deprecation_warning(macro, package, model) %}
{%- set error_message = "Warning: the `" ~ macro ~"` macro is no longer available in dbt_utils and backwards compatibility will be removed in a future version. Use `dbt." ~ macro ~ "` instead. The " ~ package ~ "." ~ model ~ " model triggered this warning." -%}
{%- do exceptions.warn(error_message) -%}
{% endmacro %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jtcohen6 and I decided here to omit the dbt namespace for the public-facing documentation for cross-database macros.

I feel the public-facing documentation and the deprecation warning should be in tight alignment; it's confusing for end users when they differ.

There are two different permissible options here (with dbt. prefix or without). I wonder if this style guide or somewhere else would be good to document our explicit recommendation for which option to use?

Suggested change
{% macro xdb_deprecation_warning(macro, package, model) %}
{%- set error_message = "Warning: the `" ~ macro ~"` macro is no longer available in dbt_utils and backwards compatibility will be removed in a future version. Use `dbt." ~ macro ~ "` instead. The " ~ package ~ "." ~ model ~ " model triggered this warning." -%}
{%- do exceptions.warn(error_message) -%}
{% endmacro %}
{% macro xdb_deprecation_warning(macro, package, model) %}
{%- set error_message = "Warning: the `" ~ macro ~"` macro is no longer available in dbt_utils and backwards compatibility will be removed in a future version. Drop the `dbt_utils.` prefix and use just `" ~ macro ~ "` instead. The " ~ package ~ "." ~ model ~ " model triggered this warning." -%}
{%- do exceptions.warn(error_message) -%}
{% endmacro %}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I 💯 agree with your desire for consistency of advice, especially for the exact same functions.

Ideally that should be true for all macros across the board, which would mean that we should have no dbt. to be in alignment with other out of the box macros (e.g it would be weird for someone to start going around saying {{ dbt.ref('my_model') }}.

BUT.

My gut feeling on why these ones are different is that they get a bit uncanny-valley-y. If I am new to dbt, then coming across {{ dateadd() }}, {{ except() }} etc could feel quite confusing - do I need to put all SQL keywords into Jinja braces? (In fairness, people shouldn't come across this in their early days, because it should all be deep in the bowels of packages' code.)

If we do {{ dbt.dateadd() }}, then it's clear that it's behaviour provided by dbt, even though it looks a bunch like standard SQL.

This sort of matches up with something else I've been thinking of, which is the shorthand node selection syntax being the same for a package or a model or a directory (dbt run -s something). I think we're doing a disservice letting the default advice be "type whatever you want! We'll work it out!" versus the more explicit package:something, path:something. But that's a different issue for a different day.

If you or jerco feel very strongly in the opposite direction, I'm willing to be talked around. I just like to know where my magic is coming from 🐰🎩

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Jerco and I both considered the two options and decided on the version that is shorter to read and write and aligns with {{ ref() }}, {{ source() }}, etc. i.e., without dbt. prefix.

Since we both had the same preference, we didn't discuss it in detail.

3 changes: 3 additions & 0 deletions macros/materializations/insert_by_period_materialization.sql
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@
{%- set start_date = config.require('start_date') -%}
{%- set stop_date = config.get('stop_date') or '' -%}
{%- set period = config.get('period') or 'week' -%}

{%- set deprecation_warning = "Warning: the `insert_by_period` materialization will be removed from dbt_utils in version 1.0.0. Install from dbt-labs/dbt-labs-experimental-features instead (see https://github.com/dbt-labs/dbt-utils/discussions/487). The " ~ package ~ "." ~ model ~ " model triggered this warning." -%}
{%- do exceptions.warn(deprecation_warning) -%}

{%- if sql.find('__PERIOD_FILTER__') == -1 -%}
{%- set error_message -%}
Expand Down