Deprecation/remove basic transformation tables

.gitignore

@@ -4,4 +4,6 @@ dbt_modules/
 logs/
 .gitignore
 .DS_Store
-dbt_packages/
+dbt_packages/
+integration_tests/.DS_Store
+integration_tests/seeds/.DS_Store

CHANGELOG.md

@@ -1,3 +1,10 @@
+# dbt_fivetran_log v1.3.0
+
+## 🚨 Breaking Changes 🚨
+- Deprecated the `transformation` and `trigger_table` source tables and any downstream transforms. These tables only housed information on Fivetran Basic SQL Transformations, which were sunset last year ([PR #96](https://github.com/fivetran/dbt_fivetran_log/pull/96)).
+  - The entire `fivetran_platform__transformation_status` end model has therefore been removed.
+  - As they are now obsolete, the `fivetran_platform_using_transformations` and `fivetran_platform_using_triggers` variables have been removed.
+
 # dbt_fivetran_log v1.2.0
 
 [PR #92](https://github.com/fivetran/dbt_fivetran_log/pull/92) includes the following updates:

README.md

@@ -17,7 +17,7 @@
 # 📣 What does this dbt package do?
 - Generates a comprehensive data dictionary of your Fivetran Platform connector (previously called Fivetran Log) data via the [dbt docs site](https://fivetran.github.io/dbt_fivetran_log/)
 - Produces staging models in the format described by [this ERD](https://fivetran.com/docs/logs/fivetran-log#schemainformation) which clean, test, and prepare your Fivetran data from [Fivetran's free connector](https://fivetran.com/docs/applications/fivetran-log) and generates analysis ready end models.
-- The above mentioned models enable you to better understand how you are spending money in Fivetran according to our [consumption-based pricing model](https://fivetran.com/docs/getting-started/consumption-based-pricing) as well as providing details about the performance and status of your Fivetran connectors and transformations. This is achieved by:
+- The above mentioned models enable you to better understand how you are spending money in Fivetran according to our [consumption-based pricing model](https://fivetran.com/docs/getting-started/consumption-based-pricing) as well as providing details about the performance and status of your Fivetran connectors. This is achieved by:
     - Displaying consumption data at the table, connector, destination, and account levels
     - Providing a history of measured free and paid monthly active rows (MAR), credit consumption, and the relationship between the two
     - Creating a history of vital daily events for each connector
@@ -30,15 +30,13 @@ Refer to the table below for a detailed view of all models materialized by defau
 | **model**                  | **description**                                                                                                                                               |
 | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | [fivetran_platform__connector_status](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__connector_status)        | Each record represents a connector loading data into a destination, enriched with data about the connector's data sync status.                                          |
-| [fivetran_platform__transformation_status](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__transformation_status)     | Each record represents a transformation, enriched with data about the transformation's last sync and any tables whose new data triggers the transformation to run. |
 | [fivetran_platform__mar_table_history](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__mar_table_history)     | Each record represents a table's free, paid, and total volume for a month, complete with data about its connector and destination.                             |
 | [fivetran_platform__usage_mar_destination_history](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__usage_mar_destination_history)    | Table of each destination's usage and active volume, per month. Includes the usage per million MAR and MAR per usage. Usage either refers to a dollar or credit amount, depending on customer's pricing model. Read more about the relationship between usage and MAR [here](https://www.fivetran.com/legal/service-consumption-table).                             |
 | [fivetran_platform__connector_daily_events](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__connector_daily_events)    | Each record represents a daily measurement of the API calls, schema changes, and record modifications made by a connector, starting from the date on which the connector was set up.                            |
 | [fivetran_platform__schema_changelog](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__schema_changelog)    | Each record represents a schema change (altering/creating tables, creating schemas, and changing schema configurations) made to a connector and contains detailed information about the schema change event.                           |
 | [fivetran_platform__audit_table](https://fivetran.github.io/dbt_fivetran_log/#!/model/model.fivetran_log.fivetran_platform__audit_table)    | Replaces the deprecated [`_fivetran_audit` table](https://fivetran.com/docs/getting-started/system-columns-and-tables#audittables). Each record represents a table being written to during a connector sync. Contains timestamps related to the connector and table-level sync progress and the sum of records inserted/replaced, updated, and deleted in the table.                             |
 <!--section-end-->
 
-
 # 🎯 How do I use the dbt package?
 ## Step 1: Pre-Requisites
 - **Connector**: Have the Fivetran Platform connector syncing data into your warehouse. 
@@ -66,7 +64,7 @@ Include the following Fivetran Platform package version range in your `packages.
 ```yaml
 packages:
   - package: fivetran/fivetran_log
-    version: [">=1.2.0", "<1.3.0"]
+    version: [">=1.3.0", "<1.4.0"]
 ```
 
 > Note that altough the source connector is now "Fivetran Platform", the package retains the old name of "fivetran_log".
@@ -81,20 +79,15 @@ vars:
 ```
 
 ## Step 4: Disable Models for Non Existent Sources
-If you have never created Fivetran-orchestrated [basic SQL transformations](https://fivetran.com/docs/transformations/basic-sql), your source data will not contain the `transformation` and `trigger_table` tables. Moreover, if you have only created *scheduled* basic transformations that are not triggered by table syncs, your source data will not contain the `trigger_table` table (though it will contain `transformation`). 
-
-Additionally, if you do not leverage Fivetran RBAC, then you will not have the `user` or `destination_membership` sources. To disable the corresponding functionality in the package, you must add the following variable(s) to your root `dbt_project.yml` file. By default, all variables are assumed to be `true`:
+If you do not leverage Fivetran RBAC, then you will not have the `user` or `destination_membership` sources. To disable the corresponding functionality in the package, you must add the following variable(s) to your root `dbt_project.yml` file. By default, all variables are assumed to be `true`:
 
 ```yml
 vars:
-    fivetran_platform_using_transformations: false # this will disable all transformation + trigger_table logic
-    fivetran_platform_using_triggers: false # this will disable only trigger_table logic 
     fivetran_platform_using_destination_membership: false # this will disable only the destination membership logic
     fivetran_platform_using_user: false # this will disable only the user logic
 ```
 
 ## (Optional) Step 5: Additional Configurations
-<details><summary>Expand for configurations</summary>
 
 ### Configuring Fivetran Error and Warning Messages
 Some users may wish to exclude Fivetran error and warnings messages from the final `fivetran_platform__connector_status` model due to the length of the message. To disable the `errors_since_last_completed_sync` and `warnings_since_last_completed_sync` fields from the final model you may add the following variable to you to your root `dbt_project.yml` file. By default, this variable is assumed to be `true`:
@@ -103,6 +96,7 @@ Some users may wish to exclude Fivetran error and warnings messages from the fin
 vars:
     fivetran_platform_using_sync_alert_messages: false # this will disable only the sync alert messages within the connector status model
 ```
+
 ### Change the Build Schema
 By default this package will build the Fivetran staging models within a schema titled (<target_schema> + `_stg_fivetran_platform`)  and the Fivetran Platform final models within your <target_schema> + `_fivetran_platform` in your target database. If this is not where you would like you Fivetran staging and final models to be written to, add the following configuration to your root `dbt_project.yml` file:
 
@@ -113,7 +107,7 @@ models:
     staging:
       +schema: my_new_staging_models_schema # leave blank for just the target_schema
 ```
-    
+
 ### Change the Source Table References
 If an individual source table has a different name than expected (see this projects [dbt_project.yml](https://github.com/fivetran/dbt_fivetran_log/blob/main/dbt_project.yml) variable declarations for expected names), provide the name of the table as it appears in your warehouse to the respective variable as identified below:
 ```yml
@@ -129,15 +123,13 @@ dispatch:
     search_order: ['spark_utils', 'dbt_utils']
 ```
 
-</details>
-
 ## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand for details</summary>
 <br>
 
 Fivetran offers the ability for you to orchestrate your dbt project through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt). Refer to the linked docs for more information on how to setup your project for orchestration through Fivetran. 
 </details>
-    
+
 # 🔍 Does this package have dependencies?
 This dbt package is dependent on the following dbt packages. Please be aware that these dependencies are installed by default within this package. For more information on the below packages, refer to the [dbt hub](https://hub.getdbt.com/) site.
 > **If you have any of these dependent packages in your own `packages.yml` I highly recommend you remove them to ensure there are no package version conflicts.**
@@ -152,7 +144,7 @@ packages:
     - package: dbt-labs/spark_utils
       version: [">=0.3.0", "<0.4.0"]
 ```
-          
+
 # 🙌 How is this package maintained and can I contribute?
 ## Package Maintenance
 The Fivetran team maintaining this package **only** maintains the latest version of the package. We highly recommend you stay consistent with the [latest version](https://hub.getdbt.com/fivetran/fivetran_log/latest/) of the package and refer to the [CHANGELOG](https://github.com/fivetran/dbt_fivetran_log/blob/main/CHANGELOG.md) and release notes for more information on changes across versions.

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'fivetran_log'
-version: '1.2.0'
+version: '1.3.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
 models:
@@ -21,7 +21,5 @@ vars:
     destination: "{{ source('fivetran_platform', 'destination') }}"
     destination_membership: "{{ source('fivetran_platform', 'destination_membership') }}"
     log: "{{ source('fivetran_platform', 'log') }}"
-    transformation: "{{ source('fivetran_platform', 'transformation') }}"
-    trigger_table: "{{ source('fivetran_platform', 'trigger_table') }}"
     user: "{{ source('fivetran_platform', 'user') }}"
     usage_cost: "{{ source('fivetran_platform', 'usage_cost') }}"

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'fivetran_log_integration_tests'
-version: '1.2.0'
+version: '1.3.0'
 
 config-version: 2
 profile: 'integration_tests'
@@ -19,8 +19,6 @@ vars:
     fivetran_platform_destination_identifier: "destination"
     fivetran_platform_destination_membership_identifier: "destination_membership"
     fivetran_platform_log_identifier: "log"
-    fivetran_platform_transformation_identifier: "transformation"
-    trigger_table: "{{ fivetran_utils.snowflake_seed_data('trigger_table') }}"
     fivetran_platform_user_identifier: "user"
 
 
@@ -57,18 +55,6 @@ seeds:
       +column_types:
         time_stamp: timestamp
         transformation_id:  "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
-    transformation:
-      +column_types:
-        created_at: timestamp
-        destination_id: "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
-        id:  "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
-    trigger_table:
-      +quote_columns: "{{ true if target.type in ('redshift', 'postgres') else false }}"
-      +enabled: "{{ true if target.type != 'snowflake' else false }}"
-      +column_types:
-        transformation_id:  "{{ 'string' if target.type in ('bigquery', 'spark', 'databricks') else 'varchar' }}"
-    trigger_table_snowflake:
-      +enabled: "{{ true if target.type == 'snowflake' else false }}"
     user:
       +column_types:
         created_at: timestamp

integration_tests/requirements.txt

@@ -4,6 +4,5 @@ dbt-redshift>=1.3.0,<2.0.0
 dbt-postgres>=1.3.0,<2.0.0
 dbt-spark>=1.3.0,<2.0.0
 dbt-spark[PyHive]>=1.3.0,<2.0.0
-dbt-databricks>=1.3.0,<2.0.0
-
+dbt-databricks>=1.6.0,<2.0.0
 oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437

models/fivetran_platform.yml

@@ -167,62 +167,6 @@ models:
       - name: dollars_spent
         description: The dollar amount used by the destination for the given month.
 
-  - name: fivetran_platform__transformation_status
-    description: >
-      Table of transformations enriched with data regarding its run status and any tables 
-      whose new data trigger the transformation to run.
-
-    tests:
-      - dbt_utils.unique_combination_of_columns:
-          combination_of_columns: 
-            - destination_id
-            - transformation_id 
-
-    columns:
-      - name: transformation_id
-        description: Unique ID of the transformation.
-
-      - name: created_at
-        description: Timestamp of when the transformation was created.
-
-      - name: created_by_user_id
-        description: Foreign key referencing the `user` who created the transformation.
-
-      - name: destination_id
-        description: Foreign key referenencing the `destination` whose data is being transformed.
-
-      - name: transformation_name
-        description: Name given to the transformation.
-
-      - name: is_paused
-        description: Boolean of whether the data transformation is currently paused.
-
-      - name: script
-        description: The code content of the transformation.
-
-      - name: trigger_delay 
-        description: The delay in triggering transformation.
-
-      - name: trigger_interval
-        description: The scheduled frequency of the transformation, if a time schedule is chosen as its trigger.
-
-      - name: trigger_type
-        description: The schedule type of the transformation (time schedule vs. new data in certain tables).
-
-      - name: destination_name
-        description: Name of the destination in which the destination is run.
-
-      - name: last_successful_run_at
-        description: Timestamp of when the transformation was last run successfully.
-
-      - name: last_run_attempt
-        description: Outcome of the last run attempt (`"success"` or `"failure"`).
-
-      - name: trigger_tables
-        description: >
-          Aggregated string list of the tables that trigger the transformation to run, if it is
-          not on a time schedule.
-
   - name: fivetran_platform__connector_daily_events
     description: >
       Table of each connector's daily history of the quantity of api calls, schema changes, and records modified,