You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Attempts to disable full-refresh for an incremental model, either by setting +full_refresh: false in dbt_project.yml or full_refresh = false in a model's config, is ignored when running dbt run --full-refresh.
Steps To Reproduce
Create a new, incremental model with the following definition:
Then run dbt run --select <model_name> --full-refresh twice.
Finally, run a select query against the table that selects the min and max value for the change timestamp.
select min(ChangeTimestamp), max(ChangeTimestamp) from <database_name>.<model_name>
This will give the same value for both the min and the max timestamp.
Expected behavior
The full-refresh config setting should be respected when the FULL_REFRESH flag is used, meaning that an incremental table with the config full_refresh = false should always load incrementally except for the first time the model is run.
Describe the bug
Attempts to disable full-refresh for an incremental model, either by setting
+full_refresh: false
in dbt_project.yml orfull_refresh = false
in a model's config, is ignored when runningdbt run --full-refresh
.Steps To Reproduce
Create a new, incremental model with the following definition:
Then run
dbt run --select <model_name> --full-refresh
twice.Finally, run a select query against the table that selects the min and max value for the change timestamp.
select min(ChangeTimestamp), max(ChangeTimestamp) from <database_name>.<model_name>
This will give the same value for both the min and the max timestamp.
Expected behavior
The full-refresh config setting should be respected when the FULL_REFRESH flag is used, meaning that an incremental table with the config full_refresh = false should always load incrementally except for the first time the model is run.
The error is caused by line 13 in incremental.sql:
https://github.com/dbt-labs/dbt-spark/blob/main/dbt/include/spark/macros/materializations/incremental/incremental.sql
{%- set full_refresh_mode = (flags.FULL_REFRESH == True) -%}
This does not respect the config definition, and does instead solely rely on the FULL_REFRESH flag.
The correct definition for this is as defined in dbt-core/dbt-snowflake/etc
https://github.com/dbt-labs/dbt-core/blob/v0.21.1rc1/core/dbt/include/global_project/macros/materializations/incremental/incremental.sql
{%- set full_refresh_mode = (should_full_refresh()) -%}
This has been tested locally and shown to work as expected.
Screenshots and log output
System information
The output of
dbt --version
:NB: the plugin that is used is dbt-spark.
The operating system you're using:
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
The output of
python --version
:Python 3.8.12
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: