Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Amendment for run_query macro #656

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

xemuliam
Copy link
Contributor

@xemuliam xemuliam commented Jan 24, 2025

Problem

Different DBT backends treat SQL query differently. For example, BigQuery can run several SQL statements as a single SQL-script.
Snowflake and Databricks can't behave in that way and require to run single SQL statement at a time.
Thus what we can do if we need to run several SQL statements?
Here you can see several options available atb the moment:

  • 1st
{{ config(
    materialized = 'table',
) }}

{%- call statement('1') -%}

  select 1

{%- endcall -%}

{%- call statement('2') -%}

  select 2

{%- endcall -%}

select 111
  • 2nd
{{ config(
    materialized = 'table',
) }}

{%- set statements = ["
    select 1
  ", "
    select 2
  "
]-%}

{% for stmt in statements %}
  {% do run_query(stmt) %}
{% endfor %}

select 111
  • 3rd
{{ config(
    materialized = 'table',
) }}

{%- set full_query %}
  select 1;
  select 2;
  select 3;

{% endset -%}

{%- set statements = full_query.split(';') -%}
{%- for stmt in statements -%}
  {%- if stmt | trim | length > 0 -%}
    {%- do run_query(stmt) -%}
  {%- endif -%}
{%- endfor -%}

select 111

Can we treat that as convenient? Probably not

Solution

Let's amend existing run_query macro to let it be executed in either single statement/script or bulk mode. Default execution mode is script. If someone wants to use it in bulk mode then the only thing needs to be added is a macro's parameter bulk=True.
Proposed change is not breaking. Not even single existing macro invocation will be jeopardized.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to meet desire functionality
  • Specific tests are not required because there is just a wrapper to existing functionality
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc)

@xemuliam xemuliam requested a review from a team as a code owner January 24, 2025 10:25
@cla-bot cla-bot bot added the cla:yes The PR author has signed the CLA label Jan 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla:yes The PR author has signed the CLA
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant