Use dbt Flags (#69)

* Use dbt Flags Signed-off-by: Yu Ishikawa <[email protected]> * Don't support dbt 1.5 or below Signed-off-by: Yu Ishikawa <[email protected]> * Edit README Signed-off-by: Yu Ishikawa <[email protected]> * Fix Signed-off-by: Yu Ishikawa <[email protected]> --------- Signed-off-by: Yu Ishikawa <[email protected]>
ubie-oss · May 29, 2024 · 8d3dfbe · 8d3dfbe
1 parent e882e53
commit 8d3dfbe
Show file tree

Hide file tree

Showing 10 changed files with 31 additions and 75 deletions.
diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml
@@ -16,57 +16,25 @@ jobs:
       matrix:
         include:
           # Python 3.8
-          - python-version: "3.8"
-            pip-requirements: "requirements-1.1.txt"
-          - python-version: "3.8"
-            pip-requirements: "requirements-1.2.txt"
-          - python-version: "3.8"
-            pip-requirements: "requirements-1.3.txt"
-          - python-version: "3.8"
-            pip-requirements: "requirements-1.4.txt"
-          - python-version: "3.8"
-            pip-requirements: "requirements-1.5.txt"
           - python-version: "3.8"
             pip-requirements: "requirements-1.6.txt"
           - python-version: "3.8"
             pip-requirements: "requirements-1.7.txt"
           # Python 3.9
-          - python-version: "3.9"
-            pip-requirements: "requirements-1.1.txt"
-          - python-version: "3.9"
-            pip-requirements: "requirements-1.2.txt"
-          - python-version: "3.9"
-            pip-requirements: "requirements-1.3.txt"
-          - python-version: "3.9"
-            pip-requirements: "requirements-1.4.txt"
-          - python-version: "3.9"
-            pip-requirements: "requirements-1.5.txt"
           - python-version: "3.9"
             pip-requirements: "requirements-1.6.txt"
           - python-version: "3.9"
             pip-requirements: "requirements-1.7.txt"
           - python-version: "3.8"
             pip-requirements: "requirements-1.8.txt"
           # Python 3.10
-          - python-version: "3.10"
-            pip-requirements: "requirements-1.1.txt"
-          - python-version: "3.10"
-            pip-requirements: "requirements-1.2.txt"
-          - python-version: "3.10"
-            pip-requirements: "requirements-1.3.txt"
-          - python-version: "3.10"
-            pip-requirements: "requirements-1.4.txt"
-          - python-version: "3.10"
-            pip-requirements: "requirements-1.5.txt"
           - python-version: "3.10"
             pip-requirements: "requirements-1.6.txt"
           - python-version: "3.10"
             pip-requirements: "requirements-1.7.txt"
           - python-version: "3.10"
             pip-requirements: "requirements-1.8.txt"
           # Python 3.11
-          - python-version: "3.11"
-            pip-requirements: "requirements-1.5.txt"
           - python-version: "3.11"
             pip-requirements: "requirements-1.6.txt"
           - python-version: "3.11"

diff --git a/README.md b/README.md
@@ -1,4 +1,5 @@
 # dbt-data-privacy
+
 This dbt package enables us to protect out customers' privacy on warehouse.
 
 <!-- toc -->
@@ -7,24 +8,27 @@ This dbt package enables us to protect out customers' privacy on warehouse.
 - [Requirements](#requirements)
 - [Supported warehouses](#supported-warehouses)
 - [Generic tests](#generic-tests)
-  * [Data Loss Prevention](#data-loss-prevention)
+  - [Data Loss Prevention](#data-loss-prevention)
 - [Macros](#macros)
-  * [Pseudonymization](#pseudonymization)
-    + [`sha256`](#sha256)
-    + [`sha512`](#sha512)
-    + [`extract_email_domain`](#extract_email_domain)
-  * [Code generation](#code-generation)
-    + [`generate_privacy_protected_models`](#generate_privacy_protected_models)
+  - [Pseudonymization](#pseudonymization)
+    - [`sha256`](#sha256)
+    - [`sha512`](#sha512)
+    - [`extract_email_domain`](#extract_email_domain)
+  - [Code generation](#code-generation)
+    - [`generate_privacy_protected_models`](#generate_privacy_protected_models)
 
 <!-- tocstop -->
 
 ## Installation Instructions
+
 COMING SOON
 
 ## Requirements
-- dbt-core: 1.1.0 or later
+
+- dbt-core: 1.6 or later
 
 ## Supported warehouses
+
 We support only BigQuery at the moment.
 But, the implementation can be extended to other warehouses by following the manner of dbt package development.
 
@@ -33,38 +37,46 @@ But, the implementation can be extended to other warehouses by following the man
 ## Generic tests
 
 ### Data Loss Prevention
+
 COMING SOON
 
 ## Macros
 
 ### Pseudonymization
 
 #### `sha256`
-Computes the hash of the input using the SHA-256 algorithm. 
+
+Computes the hash of the input using the SHA-256 algorithm.
 
 **Usage:**
+
 ```yaml
 SELECT
   {{ dbt_data_privacy.sha256("column_a") }} AS column_a,
 ```
 
 #### `sha512`
-Computes the hash of the input using the SHA-512 algorithm. 
+
+Computes the hash of the input using the SHA-512 algorithm.
 
 **Usage:**
+
 ```yaml
 SELECT
   {{ dbt_data_privacy.sha512("column_a") }} AS column_a,
 ```
 
 #### `extract_email_domain`
+
 Computes the hash of the input using the SHA-512 algorithm.
 
 **Usage:**
+
 ```yaml
 SELECT
   {{ dbt_data_privacy.extract_email_domain("email_column") }} AS email_column,
 ```
 
 ### Code generation
+
 [Generate privacy-protected dbt models](./docs/generate_models.md)
diff --git a/integration_tests/Makefile b/integration_tests/Makefile
@@ -1,5 +1,5 @@
 setup:
-	pip install --force-reinstall -r requirements/requirements-1.7.txt
+	pip install --force-reinstall -r requirements/requirements-1.8.txt
 	dbt deps --profiles-dir profiles --target bigquery
 
 

diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -20,4 +20,12 @@ clean-targets:  # directories to be removed by `dbt clean`
   - "target"
   - "dbt_modules"
 
+flags:
+  send_anonymous_usage_stats: false
+  use_colors: true
+  warn_error_options:
+    silence:
+      - ProjectFlagsMovedDeprecation
+
+
 models:
diff --git a/integration_tests/profiles/profiles.yml b/integration_tests/profiles/profiles.yml
@@ -1,8 +1,3 @@
----
-config:
-  send_anonymous_usage_stats: false
-  use_colors: true
-
 default:
   outputs:
     bigquery:

diff --git a/integration_tests/requirements/requirements-1.1.txt b/integration_tests/requirements/requirements-1.1.txt
diff --git a/integration_tests/requirements/requirements-1.2.txt b/integration_tests/requirements/requirements-1.2.txt
diff --git a/integration_tests/requirements/requirements-1.3.txt b/integration_tests/requirements/requirements-1.3.txt
diff --git a/integration_tests/requirements/requirements-1.4.txt b/integration_tests/requirements/requirements-1.4.txt
diff --git a/integration_tests/requirements/requirements-1.5.txt b/integration_tests/requirements/requirements-1.5.txt