Skip to content

Commit

Permalink
Add CONTRIBUTING.md (#80)
Browse files Browse the repository at this point in the history
Signed-off-by: Yu Ishikawa <[email protected]>
  • Loading branch information
yu-iskw committed Jan 30, 2025
1 parent 481d41e commit 5d29222
Showing 1 changed file with 84 additions and 0 deletions.
84 changes: 84 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# Contributing to dbt_data_privacy

Thank you for your interest in contributing to dbt_data_privacy!

## Before you start

We utilize a unique approach to implement unit testing for dbt packages.
This approach is necessary because Jinja2 does not inherently support unit testing.
The dbt Developer Blog provides an excellent article on implementing unit testing for dbt packages.
This article serves as a valuable resource for understanding the concept of unit testing within the context of dbt packages.

[An introduction to unit testing your dbt Packages \| dbt Developer Blog](https://docs.getdbt.com/blog/unit-testing-dbt-packages)

Unit testing becomes increasingly crucial as a dbt package grows in complexity, ensuring that the package functions as intended.

## How to develop

### Directory structure

We primarily utilize the following directories for developing the dbt package.
The `macros/` directory houses the core logic of dbt_data_privacy in the form of dbt Macros.
The `integration_tests/` directory contains test files for both unit and integration testing.

- `macros/`: Contains the dbt Macros that constitute dbt_data_privacy.
- `integration_tests/`: Houses integration tests for dbt_data_privacy, encompassing both unit and integration tests.

### How to set up the development environment

Begin by cloning the repository to your local machine.
Navigate to the `integration_tests/` directory to execute unit and integration tests.
The [./integration_tests/Makefile](./integration_tests/Makefile) provides a set of helpful commands.
The `make setup` command installs dependencies and configures the development environment.

```shell
cd dbt_data_privacy/integration_tests

# Install dependencies
make setup
```

### How to run unit testing

A dedicated Makefile target is available for running unit tests.
This target allows you to run unit tests with a specified target and a vars file.

```shell
make run-unit-tests
```

Note that unit testing cannot be parallelized due to the sequential nature of test execution.
If an error occurs in a test file, the unit testing process halts and returns an error.
Consequently, it is not possible to capture all failed tests in a single command.

### How to implement unit tests

When adding or modifying dbt Macros in the `macros/` directory, it is essential to implement corresponding unit tests.
Place the test file in the `integration_tests/macros/` directory.

For example, the dbt Macro located at [macros/codegen/generate_privacy_protected_model_sql.sql](macros/codegen/generate_privacy_protected_model_sql.sql) is responsible for generating a secured dbt Model.
The corresponding unit test for this macro can be found at [integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql](integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql).
This illustrates the convention of placing a dbt Macro and its associated unit test file in a structured manner.

Custom dbt Macros, such as `assert_equals`, are available in the [integration_tests/macros/test_utils/](integration_tests/macros/test_utils/) directory to facilitate effective unit testing of dbt Macros.
Leverage these macros to create more comprehensive and robust unit tests.

### How to run integration testing

In addition to unit testing, we also perform integration testing for dbt_data_privacy.
This involves building dbt models, including secured models generated by the dbt_data_privacy package, on a data warehouse such as BigQuery.

1. Generate secured models using the provided command.
2. Compile the dbt Models to ensure the generated secured models are valid.
3. Execute a `dbt build` command to thoroughly test the generated models.

```shell
# 1. Generate secured models
make generate

# 2. Compile dbt Models if generated secured models are valid
make compile

# 2. Run a `dbt build` command to test generated models
make run-integration-tests
```

0 comments on commit 5d29222

Please sign in to comment.