Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added suport for partition by, distribute by and primary key constraints #132

Merged
merged 4 commits into from
Jan 2, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 62 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# dbt-exasol

**[dbt](https://www.getdbt.com/)** enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Please see the dbt documentation on **[Exasol setup](https://docs.getdbt.com/reference/warehouse-setups/exasol-setup)** for more information on how to start using the Exasol adapter.

# Current profile.yml settings

<File name='profiles.yml'>

```yaml
Expand All @@ -20,10 +22,12 @@ dbt-exasol:
schema: SCHEMA
```

#### Optional login credentials using OpenID for Exasol SaaS
## Optional login credentials using OpenID for Exasol SaaS

OpenID login through access_token or refresh_token instead of user+password

#### Optional parameters
## Optional parameters

<ul>
<li><strong>connection_timeout</strong>: defaults to pyexasol default</li>
<li><strong>socket_timeout</strong>: defaults to pyexasol default</li>
Expand All @@ -37,57 +41,102 @@ OpenID login through access_token or refresh_token instead of user+password

# Known isues

## >=1.8.1 additional parameters

As of dbt-exasol 1.8.1 it is possible to add new model config parameters for models materialized as table or incremental.

<ul>
<li><strong>partition_by_config</strong></li>
<li><strong>distribute_by_config</strong></li>
<li><strong>primary_key_config</strong></li>
</ul>

- Example table materialization config

```yaml
{{
config(
materialized='table',
primary_key_config=['<column>','<column2>'],
partition_by_config='<column>',
distribute_by_config='<column>'
)
}}
```

---

**NOTE**
In case more than one column is used, put them in a list.

---

## >=1.8 license change

As of dbt-exasol version 1.8 we have decided to switch to Apache License from GPLv3 - to be equal to dbt-core licensing.

## setuptools breaking change

Due to a breaking change in setuptools and a infected dependency from dbt-core, we need to use the following [workaround for poetry install](https://github.com/pypa/setuptools/issues/4519#issuecomment-2255446798).

## Using encryption in Exasol 7 vs. 8

Starting from Exasol 8, encryption is enforced by default. If you are still using Exasol 7 and have trouble connecting, you can disable encryption in profiles.yaml (see optional parameters).

## Materialized View & Clone operations

In Exasol materialized views and clone operations are not suported. Default behaviour from dbt-core will fail accordingly.

## Null handling in test_utils null safe handling

In Exasol empty string are NULL. Due to this behaviour and as of [this pull request 7776 published in dbt-core 1.6](https://github.com/dbt-labs/dbt-core/pull/7776),
seeds in tests that use EMPTY literal to simulate empty string have to be handled with special behaviour in exasol.
See fixture for csv in exasol__seeds__data_hash_csv for tests/functional/adapter/utils/test_utils.py::TestHashExasol.
See fixture for csv in exasol**seeds**data_hash_csv for tests/functional/adapter/utils/test_utils.py::TestHashExasol.

## Model contracts

The following database constraints are implemented for Exasol:

| Constraint Type | Status |
| ------------- | ------------- |
| check | NOT supported |
| not null | enforced |
| unique | NOT supported |
| primary key | enforced |
| foreign key | enforced |
| Constraint Type | Status |
| --------------- | ------------- |
| check | NOT supported |
| not null | enforced |
| unique | NOT supported |
| primary key | enforced |
| foreign key | enforced |

## >=1.5 Incremental model update
Fallback to dbt-core implementation and supporting strategies

Fallback to dbt-core implementation and supporting strategies

- append
- merge
- delete+insert

## >=1.3 Python model not yet supported - WIP
- Please follow [this pull request](https://github.com/tglunde/dbt-exasol/pull/59)

- Please follow [this pull request](https://github.com/tglunde/dbt-exasol/pull/59)

## Breaking changes with release 1.2.2

- Timestamp format defaults to YYYY-MM-DDTHH:MI:SS.FF6

## SQL functions compatibility

### split_part

There is no equivalent SQL function in Exasol for split_part.

### listagg part_num

The SQL function listagg in Exasol does not support the num_part parameter.

## Utilities shim package
In order to support packages like dbt-utils and dbt-audit-helper, we needed to create the [shim package exasol-utils](https://github.com/tglunde/exasol-utils). In this shim package we need to adapt to parts of the SQL functionality that is not compatible with Exasol - e.g. when 'final' is being used which is a keyword in Exasol. Please visit [Adaopter dispatch documentation](https://docs.getdbt.com/guides/advanced/adapter-development/3-building-a-new-adapter#adapter-dispatch) of dbt-labs for more information.

In order to support packages like dbt-utils and dbt-audit-helper, we needed to create the [shim package exasol-utils](https://github.com/tglunde/exasol-utils). In this shim package we need to adapt to parts of the SQL functionality that is not compatible with Exasol - e.g. when 'final' is being used which is a keyword in Exasol. Please visit [Adaopter dispatch documentation](https://docs.getdbt.com/guides/advanced/adapter-development/3-building-a-new-adapter#adapter-dispatch) of dbt-labs for more information.

# Reporting bugs and contributing code

- Please report bugs using the issues

# Releases
Expand Down
2 changes: 1 addition & 1 deletion dbt/adapters/exasol/__version__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = "1.8.0"
version = "1.8.1"
8 changes: 6 additions & 2 deletions dbt/adapters/exasol/impl.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""dbt-exasol Adapter implementation extending SQLAdapter"""
from __future__ import absolute_import

from typing import Dict, Optional, List, Set, Iterable, FrozenSet, Tuple
from typing import Dict, Optional, List, Set, Iterable, FrozenSet, Tuple, Union

from itertools import chain
import agate
Expand All @@ -13,14 +13,18 @@
from dbt_common.exceptions import CompilationError
from dbt_common.utils import filter_null_values
from dbt.adapters.base.meta import available
from dbt.adapters.base.impl import ConstraintSupport
from dbt.adapters.base.impl import ConstraintSupport, AdapterConfig
from dbt_common.contracts.constraints import ConstraintType


from dbt.adapters.exasol import (ExasolColumn, ExasolConnectionManager,
ExasolRelation)


class ExasolConfig(AdapterConfig):
partition_by_config: Optional[Union[str, List[str]]] = None
distribute_by_config: Optional[Union[str, List[str]]] = None
primary_key_config: Optional[Union[str, List[str]]] = None


class ExasolAdapter(SQLAdapter):
Expand Down
13 changes: 9 additions & 4 deletions dbt/include/exasol/macros/adapters.sql
Original file line number Diff line number Diff line change
Expand Up @@ -85,18 +85,23 @@ AS

{% macro exasol__create_table_as(temporary, relation, sql) -%}
{%- set contract_config = config.get('contract') -%}

{%- set partition_by_config = config.get('partition_by_config') -%}
{%- set distribute_by_config = config.get('distribute_by_config') -%}
{%- set primary_key_config = config.get('primary_key_config') -%}
CREATE OR REPLACE TABLE {{ relation.schema }}.{{ relation.identifier }}
{%- if contract_config.enforced -%}
{{- get_assert_columns_equivalent(sql) }}
{{ get_table_columns_and_constraints() -}};
{%- set sql = get_select_subquery(sql) %}
|SEPARATEMEPLEASE|
INSERT INTO {{ relation.schema }}.{{ relation.identifier }}
{{ sql }}
{%- else %}
AS
{{ sql }}
{{ sql }} with no data;
{% endif %}
{{ add_constraints(relation, partition_by_config, distribute_by_config, primary_key_config) }}
|SEPARATEMEPLEASE|
INSERT INTO {{ relation.schema }}.{{ relation.identifier }}
{{ sql }}
{% endmacro %}

{% macro exasol__truncate_relation(relation) -%}
Expand Down
52 changes: 52 additions & 0 deletions dbt/include/exasol/macros/create_table_helpers.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
{% macro partition_by_conf(partition_by_config) %}
{%- if partition_by_config is not none and partition_by_config is string -%}
{%- set partition_by_config = [partition_by_config] -%}
{%- endif -%}
{%- if partition_by_config is not none -%}
{%- set partition_by_string = 'partition by ' ~ partition_by_config|join(", ") -%}
{% else %}
{%- set partition_by_string = '' -%}
{%- endif -%}
{{return(partition_by_string)}}
{% endmacro %}

{% macro distribute_by_conf(distribute_by_config) %}
{%- if distribute_by_config is not none and distribute_by_config is string -%}
{%- set distribute_by_config = [distribute_by_config] -%}
{%- endif -%}
{%- if distribute_by_config is not none -%}
{%- set distribute_by_string = 'distribute by ' ~ distribute_by_config|join(", ") -%}
{% else %}
{%- set distribute_by_string = '' -%}
{%- endif -%}
{{return(distribute_by_string)}}
{% endmacro %}

{% macro primary_key_conf(primary_key_config, relation) %}
{%- if primary_key_config is not none and primary_key_config is string -%}
{%- set primary_key_config = [primary_key_config] -%}
{%- endif -%}
{%- if primary_key_config is not none -%}
{%- set primary_key_string = ' add constraint ' ~relation|replace('.','_')~'__pk primary key(' ~ primary_key_config|join(", ") ~ ')' -%}
{% else %}
{%- set primary_key_string = '' -%}
{%- endif -%}
{{return(primary_key_string)}}
{% endmacro %}

{% macro add_constraints(target_relation, partition_by_config, distribute_by_config, primary_key_config) %}
{%- if partition_by_config is not none -%}
|SEPARATEMEPLEASE|
ALTER TABLE {{target_relation}} {{partition_by_conf(partition_by_config)}};
{% endif %}

{%- if distribute_by_config is not none -%}
|SEPARATEMEPLEASE|
ALTER TABLE {{target_relation}} {{distribute_by_conf(distribute_by_config)}};
{% endif %}

{%- if primary_key_config is not none -%}
|SEPARATEMEPLEASE|
ALTER TABLE {{target_relation}} {{primary_key_conf(primary_key_config, target_relation)}};
{% endif %}
{% endmacro %}
1 change: 0 additions & 1 deletion dbt/include/exasol/macros/materializations/incremental.sql
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
{% set grant_config = config.get('grants') %}
{%- set full_refresh_mode = (should_full_refresh() or existing_relation.is_view) -%}


{{ run_hooks(pre_hooks, inside_transaction=False) }}

-- `BEGIN` happens here:
Expand Down
Loading