added suport for partition by, distribute by and primary key constrai…

…nts (#132) * added suport for partition by, distribute by and primary key constraints * added explanation in Readme file * moved functionality to 'create table as' macro --------- Co-authored-by: Torsten Glunde <[email protected]>
tglunde · Jan 2, 2025 · 66213ea · 66213ea
1 parent be40353
commit 66213ea
Show file tree

Hide file tree

Showing 8 changed files with 402 additions and 283 deletions.
diff --git a/README.md b/README.md
@@ -1,9 +1,11 @@
 # dbt-exasol
+
 **[dbt](https://www.getdbt.com/)** enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.
 
 Please see the dbt documentation on **[Exasol setup](https://docs.getdbt.com/reference/warehouse-setups/exasol-setup)** for more information on how to start using the Exasol adapter.
 
 # Current profile.yml settings
+
 <File name='profiles.yml'>
 
 ```yaml
@@ -20,10 +22,12 @@ dbt-exasol:
       schema: SCHEMA
 ```
 
-#### Optional login credentials using OpenID for Exasol SaaS
+## Optional login credentials using OpenID for Exasol SaaS
+
 OpenID login through access_token or refresh_token instead of user+password
 
-#### Optional parameters
+## Optional parameters
+
 <ul>
   <li><strong>connection_timeout</strong>: defaults to pyexasol default</li>
   <li><strong>socket_timeout</strong>: defaults to pyexasol default</li>
@@ -37,57 +41,102 @@ OpenID login through access_token or refresh_token instead of user+password
 
 # Known isues
 
+## >=1.8.1 additional parameters
+
+As of dbt-exasol 1.8.1 it is possible to add new model config parameters for models materialized as table or incremental.
+
+<ul>
+<li><strong>partition_by_config</strong></li>
+<li><strong>distribute_by_config</strong></li>
+<li><strong>primary_key_config</strong></li>
+</ul>
+
+- Example table materialization config
+
+```yaml
+{{
+    config(
+        materialized='table',
+        primary_key_config=['<column>','<column2>'],
+        partition_by_config='<column>',
+        distribute_by_config='<column>'
+    )
+}}
+```
+
+---
+
+**NOTE**
+In case more than one column is used, put them in a list.
+
+---
+
 ## >=1.8 license change
+
 As of dbt-exasol version 1.8 we have decided to switch to Apache License from GPLv3 - to be equal to dbt-core licensing.
 
 ## setuptools breaking change
+
 Due to a breaking change in setuptools and a infected dependency from dbt-core, we need to use the following [workaround for poetry install](https://github.com/pypa/setuptools/issues/4519#issuecomment-2255446798).
 
 ## Using encryption in Exasol 7 vs. 8
+
 Starting from Exasol 8, encryption is enforced by default. If you are still using Exasol 7 and have trouble connecting, you can disable encryption in profiles.yaml (see optional parameters).
 
 ## Materialized View & Clone operations
+
 In Exasol materialized views and clone operations are not suported. Default behaviour from dbt-core will fail accordingly.
 
 ## Null handling in test_utils null safe handling
+
 In Exasol empty string are NULL. Due to this behaviour and as of [this pull request 7776 published in dbt-core 1.6](https://github.com/dbt-labs/dbt-core/pull/7776),
 seeds in tests that use EMPTY literal to simulate empty string have to be handled with special behaviour in exasol.
-See fixture for csv in exasol__seeds__data_hash_csv for tests/functional/adapter/utils/test_utils.py::TestHashExasol. 
+See fixture for csv in exasol**seeds**data_hash_csv for tests/functional/adapter/utils/test_utils.py::TestHashExasol.
 
 ## Model contracts
+
 The following database constraints are implemented for Exasol:
 
-| Constraint Type  | Status |
-| ------------- | ------------- |
-| check | NOT supported  |
-| not null  | enforced  |
-| unique  | NOT supported  |
-| primary key  | enforced  |
-| foreign key  | enforced  |
+| Constraint Type | Status        |
+| --------------- | ------------- |
+| check           | NOT supported |
+| not null        | enforced      |
+| unique          | NOT supported |
+| primary key     | enforced      |
+| foreign key     | enforced      |
 
 ## >=1.5 Incremental model update
-Fallback to dbt-core implementation and supporting strategies 
+
+Fallback to dbt-core implementation and supporting strategies
+
 - append
 - merge
 - delete+insert
 
 ## >=1.3 Python model not yet supported - WIP
-- Please follow [this pull request](https://github.com/tglunde/dbt-exasol/pull/59) 
+
+- Please follow [this pull request](https://github.com/tglunde/dbt-exasol/pull/59)
 
 ## Breaking changes with release 1.2.2
+
 - Timestamp format defaults to YYYY-MM-DDTHH:MI:SS.FF6
 
 ## SQL functions compatibility
 
 ### split_part
+
 There is no equivalent SQL function in Exasol for split_part.
 
 ### listagg part_num
+
 The SQL function listagg in Exasol does not support the num_part parameter.
 
 ## Utilities shim package
-In order to support packages like dbt-utils and dbt-audit-helper, we needed to create the [shim package exasol-utils](https://github.com/tglunde/exasol-utils). In this shim package we need to adapt to parts of the SQL functionality that is not compatible with Exasol - e.g. when 'final' is being used which is a keyword in Exasol. Please visit [Adaopter dispatch documentation](https://docs.getdbt.com/guides/advanced/adapter-development/3-building-a-new-adapter#adapter-dispatch) of dbt-labs for more information. 
+
+In order to support packages like dbt-utils and dbt-audit-helper, we needed to create the [shim package exasol-utils](https://github.com/tglunde/exasol-utils). In this shim package we need to adapt to parts of the SQL functionality that is not compatible with Exasol - e.g. when 'final' is being used which is a keyword in Exasol. Please visit [Adaopter dispatch documentation](https://docs.getdbt.com/guides/advanced/adapter-development/3-building-a-new-adapter#adapter-dispatch) of dbt-labs for more information.
+
 # Reporting bugs and contributing code
+
 - Please report bugs using the issues
 
 # Releases

diff --git a/dbt/adapters/exasol/__version__.py b/dbt/adapters/exasol/__version__.py
@@ -1 +1 @@
-version = "1.8.0"
+version = "1.8.1"
diff --git a/dbt/adapters/exasol/impl.py b/dbt/adapters/exasol/impl.py
@@ -1,7 +1,7 @@
 """dbt-exasol Adapter implementation extending SQLAdapter"""
 from __future__ import absolute_import
 
-from typing import Dict, Optional, List, Set, Iterable, FrozenSet, Tuple
+from typing import Dict, Optional, List, Set, Iterable, FrozenSet, Tuple, Union
 
 from itertools import chain
 import agate
@@ -13,14 +13,18 @@
 from dbt_common.exceptions import CompilationError
 from dbt_common.utils import filter_null_values
 from dbt.adapters.base.meta import available
-from dbt.adapters.base.impl import ConstraintSupport
+from dbt.adapters.base.impl import ConstraintSupport, AdapterConfig
 from dbt_common.contracts.constraints import ConstraintType
 
 
 from dbt.adapters.exasol import (ExasolColumn, ExasolConnectionManager,
                                  ExasolRelation)
 
 
+class ExasolConfig(AdapterConfig):
+    partition_by_config: Optional[Union[str, List[str]]] = None
+    distribute_by_config: Optional[Union[str, List[str]]] = None
+    primary_key_config: Optional[Union[str, List[str]]] = None
 
 
 class ExasolAdapter(SQLAdapter):

diff --git a/dbt/include/exasol/macros/adapters.sql b/dbt/include/exasol/macros/adapters.sql
@@ -85,18 +85,23 @@ AS
 
 {% macro exasol__create_table_as(temporary, relation, sql) -%}
     {%- set contract_config = config.get('contract') -%}
+
+    {%- set partition_by_config = config.get('partition_by_config') -%}
+    {%- set distribute_by_config = config.get('distribute_by_config') -%}
+    {%- set primary_key_config = config.get('primary_key_config') -%}
     CREATE OR REPLACE TABLE {{ relation.schema }}.{{ relation.identifier }} 
         {%- if contract_config.enforced -%}
           {{- get_assert_columns_equivalent(sql) }}
           {{ get_table_columns_and_constraints() -}};
           {%- set sql = get_select_subquery(sql) %}
-        |SEPARATEMEPLEASE|
-        INSERT INTO {{ relation.schema }}.{{ relation.identifier }}
-        {{ sql }}   
         {%- else %}
         AS 
-        {{ sql }}
+        {{ sql }} with no data;
         {% endif %}
+    {{ add_constraints(relation, partition_by_config, distribute_by_config, primary_key_config) }}
+  |SEPARATEMEPLEASE|
+  INSERT INTO {{ relation.schema }}.{{ relation.identifier }}
+        {{ sql }}
 {% endmacro %}
 
 {% macro exasol__truncate_relation(relation) -%}

diff --git a/dbt/include/exasol/macros/create_table_helpers.sql b/dbt/include/exasol/macros/create_table_helpers.sql
@@ -0,0 +1,52 @@
+{% macro partition_by_conf(partition_by_config) %} 
+    {%- if partition_by_config is not none and partition_by_config is string -%}
+        {%- set partition_by_config = [partition_by_config] -%}
+    {%- endif -%}
+    {%- if partition_by_config is not none -%}
+        {%- set partition_by_string = 'partition by ' ~ partition_by_config|join(", ") -%}
+    {% else %}
+        {%- set partition_by_string = '' -%}
+    {%- endif -%}
+    {{return(partition_by_string)}}
+{% endmacro %}
+
+{% macro distribute_by_conf(distribute_by_config) %} 
+    {%- if distribute_by_config is not none and distribute_by_config is string -%}
+        {%- set distribute_by_config = [distribute_by_config] -%}
+    {%- endif -%}
+    {%- if distribute_by_config is not none -%}
+        {%- set distribute_by_string = 'distribute by ' ~ distribute_by_config|join(", ") -%}
+    {% else %}
+        {%- set distribute_by_string = '' -%}
+    {%- endif -%}
+    {{return(distribute_by_string)}}
+{% endmacro %}
+
+{% macro primary_key_conf(primary_key_config, relation) %} 
+    {%- if primary_key_config is not none and primary_key_config is string -%}
+        {%- set primary_key_config = [primary_key_config] -%}
+    {%- endif -%}
+    {%- if primary_key_config is not none -%}
+        {%- set primary_key_string = ' add constraint ' ~relation|replace('.','_')~'__pk primary key(' ~ primary_key_config|join(", ") ~ ')' -%}
+    {% else %}
+        {%- set primary_key_string = '' -%}
+    {%- endif -%}
+    {{return(primary_key_string)}}
+{% endmacro %}
+
+{% macro add_constraints(target_relation, partition_by_config, distribute_by_config, primary_key_config) %}
+    {%- if partition_by_config is not none -%}
+        |SEPARATEMEPLEASE|
+            ALTER TABLE {{target_relation}} {{partition_by_conf(partition_by_config)}};
+    {% endif %}
+
+    {%- if distribute_by_config is not none -%}
+        |SEPARATEMEPLEASE|
+            ALTER TABLE {{target_relation}} {{distribute_by_conf(distribute_by_config)}}; 
+    {% endif %}
+
+    {%- if primary_key_config is not none -%}
+        |SEPARATEMEPLEASE|
+            ALTER TABLE {{target_relation}} {{primary_key_conf(primary_key_config, target_relation)}};
+    {% endif %}  
+{% endmacro %}
diff --git a/dbt/include/exasol/macros/materializations/incremental.sql b/dbt/include/exasol/macros/materializations/incremental.sql
@@ -9,7 +9,6 @@
   {% set  grant_config = config.get('grants') %}
   {%- set full_refresh_mode = (should_full_refresh()  or existing_relation.is_view) -%}
 
-
   {{ run_hooks(pre_hooks, inside_transaction=False) }}
 
   -- `BEGIN` happens here: