From af1bad2253e11181b8c21f4e5f2c84a9b93eb659 Mon Sep 17 00:00:00 2001 From: Ly Nguyen <ly.nguyen@dbtlabs.com> Date: Thu, 13 Jul 2023 14:52:01 -0700 Subject: [PATCH] New empty mode for indirect selection --- .../node-selection/test-selection-examples.md | 82 ++++++++++++++++++- .../node-selection/yaml-selectors.md | 54 +++++++++++- 2 files changed, 130 insertions(+), 6 deletions(-) diff --git a/website/docs/reference/node-selection/test-selection-examples.md b/website/docs/reference/node-selection/test-selection-examples.md index 6a884b2035d..7e79754ed31 100644 --- a/website/docs/reference/node-selection/test-selection-examples.md +++ b/website/docs/reference/node-selection/test-selection-examples.md @@ -58,7 +58,7 @@ The "cautious" mode can be useful in environments when you're only building a su </VersionBlock> -<VersionBlock firstVersion="1.4"> +<VersionBlock firstVersion="1.4" lastVersion="1.4"> There are three modes to configure the behavior when performing indirect selection (with `eager` as the default): @@ -72,6 +72,21 @@ The "buildable" and "cautious" modes can be useful in environments when you're o </VersionBlock> +<VersionBlock firstVersion="1.5" > + +There are three modes to configure the behavior when performing indirect selection (with `eager` as the default): + +1. `eager` (default) - include ANY test that references the selected nodes +1. `cautious` - restrict to tests that ONLY refer to selected nodes +1. `buildable` - restrict to tests that ONLY refer to selected nodes (or their ancestors) +1. `empty` - restrict to tests that are only for the selected node and ignore all tests from the attached nodes + +Note that test exclusion is always greedy: if ANY parent is explicitly excluded, the test will be excluded as well. + +The "buildable", "cautious", and "empty" modes can be useful in environments when you're only building a subset of your DAG, and you want to avoid test failures in "eager" mode caused by unbuilt resources. (Another way to achieve this is with [deferral](/reference/node-selection/defer)). + +</VersionBlock> + <!--tabs for eager mode, cautious mode, and buildable mode --> <VersionBlock lastVersion="1.3"> @@ -109,7 +124,57 @@ $ dbt build --select orders --indirect-selection=cautious </VersionBlock> -<VersionBlock firstVersion="1.4"> +<VersionBlock firstVersion="1.4" lastVersion="1.4"> + +<Tabs queryString="indirect-selection-mode"> +<TabItem value="eager" label="Eager mode (default)"> + +By default, a test will run when ANY parent is selected; we call this "eager" indirect selection. In this example, that would include any test that references orders, even if it references other models as well. + +In this mode, any test that depends on unbuilt resources will raise an error. + +```shell +$ dbt test --select orders +$ dbt build --select orders +``` + +</TabItem> + +<TabItem value="cautious" label="Cautious mode"> + +It is possible to prevent tests from running if one or more of its parents is unselected (and therefore unbuilt); we call this "cautious" indirect selection. + +It will only include tests whose references are each within the selected nodes. + +Put another way, it will prevent tests from running if one or more of its parents is unselected. + +```shell +$ dbt test --select orders --indirect-selection=cautious +$ dbt build --select orders --indirect-selection=cautious +``` + +</TabItem> + +<TabItem value="buildable" label="Buildable mode"> + +This mode is similarly conservative like "cautious", but is slightly more inclusive. + +It will only include tests whose references are each within the selected nodes (or their ancestors). + +This is useful in the same scenarios as "cautious", but also includes when a test depends on a model **and** a direct ancestor of that model (like confirming an aggregation has the same totals as its input). + +```shell +$ dbt test --select orders --indirect-selection=buildable +$ dbt build --select orders --indirect-selection=buildable +``` + +</TabItem> + +</Tabs> + +</VersionBlock> + +<VersionBlock firstVersion="1.5"> <Tabs queryString="indirect-selection-mode"> <TabItem value="eager" label="Eager mode (default)"> @@ -155,11 +220,22 @@ $ dbt build --select orders --indirect-selection=buildable </TabItem> +<TabItem value="empty" label="Empty mode"> + +This mode will only include tests whose references are each within the selected nodes and will ignore all tests from attached nodes. + +```shell +$ dbt test --select orders --indirect-selection=empty +$ dbt build --select orders --indirect-selection=empty +``` + +</TabItem> + </Tabs> </VersionBlock> -<!--End of tabs for eager mode, cautious mode, and buildable mode --> +<!--End of tabs for eager mode, cautious mode, buildable mode, and empty mode --> ### Syntax examples diff --git a/website/docs/reference/node-selection/yaml-selectors.md b/website/docs/reference/node-selection/yaml-selectors.md index 95c3c791b53..78342e32779 100644 --- a/website/docs/reference/node-selection/yaml-selectors.md +++ b/website/docs/reference/node-selection/yaml-selectors.md @@ -78,7 +78,7 @@ definition: </VersionBlock> -<VersionBlock firstVersion="1.4"> +<VersionBlock firstVersion="1.4" lastVersion="1.4"> ```yml definition: @@ -95,7 +95,29 @@ definition: childrens_parents: true | false # @ operator - indirect_selection: eager | cautious | buildable # include all tests selected indirectly? eager by default + indirect_selection: eager | cautious | buildable # include all tests selected indirectly? eager by default +``` + +</VersionBlock> + +<VersionBlock firstVersion="1.5"> + +```yml +definition: + method: tag + value: nightly + + # Optional keywords map to the `+` and `@` graph operators: + + children: true | false + parents: true | false + + children_depth: 1 # if children: true, degrees to include + parents_depth: 1 # if parents: true, degrees to include + + childrens_parents: true | false # @ operator + + indirect_selection: eager | cautious | buildable | empty # include all tests selected indirectly? eager by default ``` </VersionBlock> @@ -168,7 +190,7 @@ If provided, a YAML selector's `indirect_selection` value will take precedence o </VersionBlock> -<VersionBlock firstVersion="1.4"> +<VersionBlock firstVersion="1.4" lastVersion="1.4"> As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious" or "buildable" by setting `indirect_selection` for a specific criterion: @@ -191,6 +213,32 @@ If provided, a YAML selector's `indirect_selection` value will take precedence o </VersionBlock> +<VersionBlock firstVersion="1.5"> + +As a general rule, dbt will indirectly select _all_ tests if they touch _any_ resource that you're selecting directly. We call this "eager" indirect selection. You can optionally switch the indirect selection mode to "cautious", "buildable", or "empty" by setting `indirect_selection` for a specific criterion: + +```yml +- union: + - method: fqn + value: model_a + indirect_selection: eager # default: will include all tests that touch model_a + - method: fqn + value: model_b + indirect_selection: cautious # will not include tests touching model_b + # if they have other unselected parents + - method: fqn + value: model_c + indirect_selection: buildable # will not include tests touching model_c + # if they have other unselected parents (unless they have an ancestor that is selected) + - method: fqn + value: model_d + indirect_selection: empty # will include tests for only the selected node and ignore all tests attached to model_d +``` + +If provided, a YAML selector's `indirect_selection` value will take precedence over the CLI flag `--indirect-selection`. Because `indirect_selection` is defined separately for _each_ selection criterion, it's possible to mix eager/cautious/buildable/empty modes within the same definition, to achieve the exact behavior that you need. Remember that you can always test out your critiera with `dbt ls --selector`. + +</VersionBlock> + See [test selection examples](/reference/node-selection/test-selection-examples) for more details about indirect selection. ## Example