-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: Yu Ishikawa <[email protected]>
- Loading branch information
Showing
1 changed file
with
84 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,84 @@ | ||
# Contributing to dbt_data_privacy | ||
|
||
Thank you for your interest in contributing to dbt_data_privacy! | ||
|
||
## Before you start | ||
|
||
We utilize a unique approach to implement unit testing for dbt packages. | ||
This approach is necessary because Jinja2 does not inherently support unit testing. | ||
The dbt Developer Blog provides an excellent article on implementing unit testing for dbt packages. | ||
This article serves as a valuable resource for understanding the concept of unit testing within the context of dbt packages. | ||
|
||
[An introduction to unit testing your dbt Packages \| dbt Developer Blog](https://docs.getdbt.com/blog/unit-testing-dbt-packages) | ||
|
||
Unit testing becomes increasingly crucial as a dbt package grows in complexity, ensuring that the package functions as intended. | ||
|
||
## How to develop | ||
|
||
### Directory structure | ||
|
||
We primarily utilize the following directories for developing the dbt package. | ||
The `macros/` directory houses the core logic of dbt_data_privacy in the form of dbt Macros. | ||
The `integration_tests/` directory contains test files for both unit and integration testing. | ||
|
||
- `macros/`: Contains the dbt Macros that constitute dbt_data_privacy. | ||
- `integration_tests/`: Houses integration tests for dbt_data_privacy, encompassing both unit and integration tests. | ||
|
||
### How to set up the development environment | ||
|
||
Begin by cloning the repository to your local machine. | ||
Navigate to the `integration_tests/` directory to execute unit and integration tests. | ||
The [./integration_tests/Makefile](./integration_tests/Makefile) provides a set of helpful commands. | ||
The `make setup` command installs dependencies and configures the development environment. | ||
|
||
```shell | ||
cd dbt_data_privacy/integration_tests | ||
|
||
# Install dependencies | ||
make setup | ||
``` | ||
|
||
### How to run unit testing | ||
|
||
A dedicated Makefile target is available for running unit tests. | ||
This target allows you to run unit tests with a specified target and a vars file. | ||
|
||
```shell | ||
make run-unit-tests | ||
``` | ||
|
||
Note that unit testing cannot be parallelized due to the sequential nature of test execution. | ||
If an error occurs in a test file, the unit testing process halts and returns an error. | ||
Consequently, it is not possible to capture all failed tests in a single command. | ||
|
||
### How to implement unit tests | ||
|
||
When adding or modifying dbt Macros in the `macros/` directory, it is essential to implement corresponding unit tests. | ||
Place the test file in the `integration_tests/macros/` directory. | ||
|
||
For example, the dbt Macro located at [macros/codegen/generate_privacy_protected_model_sql.sql](macros/codegen/generate_privacy_protected_model_sql.sql) is responsible for generating a secured dbt Model. | ||
The corresponding unit test for this macro can be found at [integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql](integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql). | ||
This illustrates the convention of placing a dbt Macro and its associated unit test file in a structured manner. | ||
|
||
Custom dbt Macros, such as `assert_equals`, are available in the [integration_tests/macros/test_utils/](integration_tests/macros/test_utils/) directory to facilitate effective unit testing of dbt Macros. | ||
Leverage these macros to create more comprehensive and robust unit tests. | ||
|
||
### How to run integration testing | ||
|
||
In addition to unit testing, we also perform integration testing for dbt_data_privacy. | ||
This involves building dbt models, including secured models generated by the dbt_data_privacy package, on a data warehouse such as BigQuery. | ||
|
||
1. Generate secured models using the provided command. | ||
2. Compile the dbt Models to ensure the generated secured models are valid. | ||
3. Execute a `dbt build` command to thoroughly test the generated models. | ||
|
||
```shell | ||
# 1. Generate secured models | ||
make generate | ||
|
||
# 2. Compile dbt Models if generated secured models are valid | ||
make compile | ||
|
||
# 2. Run a `dbt build` command to test generated models | ||
make run-integration-tests | ||
``` |