Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to applied state metadata #5089

Merged
merged 71 commits into from
Mar 27, 2024
Merged
Show file tree
Hide file tree
Changes from 66 commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
a582ff6
Improvements to applied state metadata
nghi-ly Mar 15, 2024
667c6e9
Minor nit
nghi-ly Mar 15, 2024
d7ecd3f
Merge branch 'current' into ly-docs-issue-5010
nghi-ly Mar 18, 2024
207926e
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
7eebf0d
This branch was auto-updated!
github-actions[bot] Mar 18, 2024
93e96a6
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
d547124
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
c32e8e1
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
47a2a0f
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
244a8ea
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
486b47d
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
26ebdc3
This branch was auto-updated!
github-actions[bot] Mar 19, 2024
d4c6e68
This branch was auto-updated!
github-actions[bot] Mar 20, 2024
9f39276
This branch was auto-updated!
github-actions[bot] Mar 20, 2024
662a01c
This branch was auto-updated!
github-actions[bot] Mar 20, 2024
2dd8b73
This branch was auto-updated!
github-actions[bot] Mar 20, 2024
de4cc46
This branch was auto-updated!
github-actions[bot] Mar 20, 2024
fb69b60
This branch was auto-updated!
github-actions[bot] Mar 20, 2024
113761a
Feedback
nghi-ly Mar 21, 2024
c580715
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
ed2d85d
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
393fb53
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
3109dda
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
8b18e1d
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
b3c9fb9
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
1f60184
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
c8200d6
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
f4525cc
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
2ef9c83
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
81337f1
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
95bbd2f
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
1df39a5
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
721dc66
This branch was auto-updated!
github-actions[bot] Mar 21, 2024
5c3d738
This branch was auto-updated!
github-actions[bot] Mar 22, 2024
9f3e362
This branch was auto-updated!
github-actions[bot] Mar 22, 2024
c3f8941
This branch was auto-updated!
github-actions[bot] Mar 22, 2024
c6ba349
This branch was auto-updated!
github-actions[bot] Mar 22, 2024
f22a68b
This branch was auto-updated!
github-actions[bot] Mar 22, 2024
e7308c7
This branch was auto-updated!
github-actions[bot] Mar 22, 2024
22eff2b
This branch was auto-updated!
github-actions[bot] Mar 23, 2024
2a414c3
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
db0befd
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
f9ace12
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
3d5e1dc
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
3aa051f
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
8b98e71
Feedback
nghi-ly Mar 25, 2024
a135c8e
Merge branch 'ly-docs-issue-5010' of github.com:dbt-labs/docs.getdbt.…
nghi-ly Mar 25, 2024
6454029
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
aaaabb9
This branch was auto-updated!
github-actions[bot] Mar 25, 2024
0c962ab
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
c2ed6c2
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
a9dbd96
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
75f75d2
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
0e94bf1
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
02fa396
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
c0e1fa0
This branch was auto-updated!
github-actions[bot] Mar 26, 2024
8b8c7ed
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
d19334f
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
8e9a6a6
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
a940622
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
51ddd01
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
28ff0f7
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
d92facf
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
6e4d2fe
This branch was auto-updated!
github-actions[bot] Mar 27, 2024
740ee75
PM feedback
nghi-ly Mar 27, 2024
8868660
Feedback
nghi-ly Mar 27, 2024
2688dec
Update website/docs/docs/core/connect-data-platform/about-core-connec…
nghi-ly Mar 27, 2024
276fd06
Update website/docs/docs/core/connect-data-platform/about-core-connec…
nghi-ly Mar 27, 2024
6a5f96c
Feedback
nghi-ly Mar 27, 2024
a3d2c97
Update website/docs/docs/core/connect-data-platform/about-core-connec…
nghi-ly Mar 27, 2024
9a318a1
Update website/docs/docs/core/connect-data-platform/about-core-connec…
nghi-ly Mar 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,30 @@ These connection instructions provide the basic fields required for configuring
If you're using dbt from the command line (CLI), you'll need a profiles.yml file that contains the connection details for your data platform. When you run dbt from the CLI, it reads your dbt_project.yml file to find the profile name, and then looks for a profile with the same name in your profiles.yml file. This profile contains all the information dbt needs to connect to your data platform.

For detailed info, you can refer to the [Connection profiles](/docs/core/connect-data-platform/connection-profiles).

<VersionBlock firstVersion="1.7">

## Adapter features
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved

The following table lists the features available for adapters:

| Adapter | Catalog | Source freshness |
|---------|---------|------------------|
| dbt default configuration | manual run | `loaded_at` field |
| `dbt-bigquery` | incremental | metadata-based and `loaded_at` field |
| `dbt-databricks` | manual run | metadata-based and `loaded_at` field |
| `dbt-postgres` | incremental | `loaded_at` field |
| `dbt-redshift` | incremental | metadata-based and `loaded_at` field |
| `dbt-snowflake` | incremental | metadata-based and `loaded_at` field |
| `dbt-spark` | manual run | `loaded_at` field |
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved


### Catalog

For adapters that support it, you can partially build the catalog. This allows for the catalog to be built along with the model, eliminating the need to run a lengthy `dbt docs generate --select ...` at the end of a dbt run. For adapters that don't support incremental catalog generation, you must run `dbt docs generate --select ...` to build the catalog.
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved

### Source freshness
You can measure source freshness using the warehouse metadata tables when the adapter supports it. This allows for calculating source freshness without using the `loaded_at` field and without querying the table directly. This is faster and more flexible. You can override this with the `loaded_at` field in the model config. If the adapter doesn't support this, you can still use the `loaded_at` field.
You can measure source freshness using the metadata when the adapter supports it. This allows for calculating source freshness without using the `loaded_at` field and without querying the table directly. This is faster and more flexible (though it might be inaccurate at times). You can override this with the `loaded_at` field in the model config. If the adapter doesn't support this, you can still use the `loaded_at` field.

</VersionBlock>
6 changes: 3 additions & 3 deletions website/docs/docs/dbt-cloud-apis/project-state.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ There are two states that can be queried in dbt Cloud:

- **Definition state** depends on what exists in the project given the code defined in it (for example, manifest state), which hasn’t necessarily been executed in the data platform (maybe just the result of `dbt compile`).

### Definition (logical) vs. applied state of dbt nodes
## Definition (logical) vs. applied state of dbt nodes
nghi-ly marked this conversation as resolved.
Show resolved Hide resolved

In a dbt project, the state of a node _definition_ represents the configuration, transformations, and dependencies defined in the SQL and YAML files. It captures how the node should be processed in relation to other nodes and tables in the data warehouse and may be produced by a `dbt build`, `run`, `parse`, or `compile`. It changes whenever the project code changes.

Expand Down Expand Up @@ -57,7 +57,7 @@ query Compare($environmentId: Int!, $first: Int!) {

Most Discovery API use cases will favor the _applied state_ since it pertains to what has actually been run and can be analyzed.

### Affected states by node type
## Affected states by node type

| Node | Executed in DAG | Created by execution | Exists in database | Lineage | States |
|-----------|------------------|----------------------|--------------------|-----------------------|----------------------|
Expand All @@ -72,7 +72,7 @@ Most Discovery API use cases will favor the _applied state_ since it pertains to
| Group | No | No | No | Downstream | Definition |
| Macro | Yes | No | No | N/A | Definition |

### Caveats about state/metadata updates
## Caveats about state/metadata updates

Over time, Cloud Artifacts will provide information to maintain state for features/services in dbt Cloud and enable you to access state in dbt Cloud and its downstream ecosystem. Cloud Artifacts is currently focused on the latest production state, but this focus will evolve.

Expand Down
Loading