Skip to content

Latest commit

 

History

History
84 lines (57 loc) · 3.85 KB

CONTRIBUTING.md

File metadata and controls

84 lines (57 loc) · 3.85 KB

Contributing to dbt_data_privacy

Thank you for your interest in contributing to dbt_data_privacy!

Before you start

We utilize a unique approach to implement unit testing for dbt packages. This approach is necessary because Jinja2 does not inherently support unit testing. The dbt Developer Blog provides an excellent article on implementing unit testing for dbt packages. This article serves as a valuable resource for understanding the concept of unit testing within the context of dbt packages.

An introduction to unit testing your dbt Packages | dbt Developer Blog

Unit testing becomes increasingly crucial as a dbt package grows in complexity, ensuring that the package functions as intended.

How to develop

Directory structure

We primarily utilize the following directories for developing the dbt package. The macros/ directory houses the core logic of dbt_data_privacy in the form of dbt Macros. The integration_tests/ directory contains test files for both unit and integration testing.

  • macros/: Contains the dbt Macros that constitute dbt_data_privacy.
  • integration_tests/: Houses integration tests for dbt_data_privacy, encompassing both unit and integration tests.

How to set up the development environment

Begin by cloning the repository to your local machine. Navigate to the integration_tests/ directory to execute unit and integration tests. The ./integration_tests/Makefile provides a set of helpful commands. The make setup command installs dependencies and configures the development environment.

cd dbt_data_privacy/integration_tests

# Install dependencies
make setup

How to run unit testing

A dedicated Makefile target is available for running unit tests. This target allows you to run unit tests with a specified target and a vars file.

make run-unit-tests

Note that unit testing cannot be parallelized due to the sequential nature of test execution. If an error occurs in a test file, the unit testing process halts and returns an error. Consequently, it is not possible to capture all failed tests in a single command.

How to implement unit tests

When adding or modifying dbt Macros in the macros/ directory, it is essential to implement corresponding unit tests. Place the test file in the integration_tests/macros/ directory.

For example, the dbt Macro located at macros/codegen/generate_privacy_protected_model_sql.sql is responsible for generating a secured dbt Model. The corresponding unit test for this macro can be found at integration_tests/macros/tests/codegen/test_generate_privacy_protected_model_sql.sql. This illustrates the convention of placing a dbt Macro and its associated unit test file in a structured manner.

Custom dbt Macros, such as assert_equals, are available in the integration_tests/macros/test_utils/ directory to facilitate effective unit testing of dbt Macros. Leverage these macros to create more comprehensive and robust unit tests.

How to run integration testing

In addition to unit testing, we also perform integration testing for dbt_data_privacy. This involves building dbt models, including secured models generated by the dbt_data_privacy package, on a data warehouse such as BigQuery.

  1. Generate secured models using the provided command.
  2. Compile the dbt Models to ensure the generated secured models are valid.
  3. Execute a dbt build command to thoroughly test the generated models.
# 1. Generate secured models
make generate

# 2. Compile dbt Models if generated secured models are valid
make compile

# 2. Run a `dbt build` command to test generated models
make run-integration-tests