1st Run
=======
2021-08-22 22:06:49.782181 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: /* {"app": "dbt", "dbt_version": "0.19.1", "profile_name": "default", "target_name": "dev", "node_id": "snapshot.test_snapshot_model.snapshot_table"} */
      create or replace table test_schema.snapshot_table
    using delta
    as
    select *,
        md5(coalesce(cast(pk_col as string ), '')
         || '|' || coalesce(cast(ts_col as string ), '')
        ) as dbt_scd_id,
        ts_col as dbt_updated_at,
        ts_col as dbt_valid_from,
        nullif(ts_col, ts_col) as dbt_valid_to
    from (
select * from test_schema.source_table
    ) sbq
2021-08-22 22:06:54.154300 (Thread-1): Poll status: 2, query complete
2021-08-22 22:06:54.155781 (Thread-1): SQL status: OK in 4.37 seconds
2021-08-22 22:06:54.166949 (Thread-1): NotImplemented: commit
2021-08-22 22:06:54.169967 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: ROLLBACK
2021-08-22 22:06:54.171198 (Thread-1): NotImplemented: rollback
2021-08-22 22:06:54.172367 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: Close
2021-08-22 22:06:54.209596 (Thread-1): Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '57edc193-91b6-457e-bfef-8703958774b8', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f4726b459d0>]}
2021-08-22 22:06:54.211440 (Thread-1): 18:06:54 | 1 of 1 OK snapshotted test_schema.snapshot_table........ [OK in 5.03s]
2021-08-22 22:06:54.212910 (Thread-1): Finished running node snapshot.test_snapshot_model.snapshot_table
2021-08-22 22:06:54.309695 (MainThread): Acquiring new spark connection "master".
2021-08-22 22:06:54.311947 (MainThread): On master: ROLLBACK
2021-08-22 22:06:54.313698 (MainThread): Opening a new connection, currently in state init
2021-08-22 22:06:54.605556 (MainThread): NotImplemented: rollback
2021-08-22 22:06:54.607184 (MainThread): NotImplemented: add_begin_query
2021-08-22 22:06:54.608673 (MainThread): NotImplemented: commit
2021-08-22 22:06:54.610140 (MainThread): On master: ROLLBACK
2021-08-22 22:06:54.611424 (MainThread): NotImplemented: rollback
2021-08-22 22:06:54.612656 (MainThread): On master: Close
2021-08-22 22:06:54.614499 (MainThread): 18:06:54 |
2021-08-22 22:06:54.615590 (MainThread): 18:06:54 | Finished running 1 snapshot in 8.27s.
2021-08-22 22:06:54.616771 (MainThread): Connection 'master' was properly closed.
2021-08-22 22:06:54.617883 (MainThread): Connection 'list_None_dbt' was properly closed.
2021-08-22 22:06:54.619088 (MainThread): Connection 'snapshot.test_snapshot_model.snapshot_table' was properly closed.
2021-08-22 22:06:54.652748 (MainThread):
2021-08-22 22:06:54.654534 (MainThread): Completed successfully
2021-08-22 22:06:54.655713 (MainThread):
Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2021-08-22 22:06:54.657098 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f4726a0de10>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f47271b8a50>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7f4726b3be50>]}
2021-08-22 22:06:54.658634 (MainThread): Flushing usage events
2021-08-22 22:06:54.753366 (MainThread): Error sending message, disabling tracking

2nd Run
=======

    Before Fix
    ==========

    021-08-22 22:07:32.936730 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: /* {"app": "dbt", "dbt_version": "0.19.1", "profile_name": "default", "target_name": "dev", "node_id": "snapshot.test_snapshot_model.snapshot_table"} */

        show databases

    2021-08-22 22:07:32.938180 (Thread-1): Opening a new connection, currently in state closed
    2021-08-22 22:07:33.298879 (Thread-1): Poll status: 2, query complete
    2021-08-22 22:07:33.300231 (Thread-1): SQL status: OK in 0.36 seconds
    2021-08-22 22:07:33.429600 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: Close
    2021-08-22 22:07:33.465756 (Thread-1): Compilation Error in snapshot snapshot_table (snapshots/snapshot_table.sql)
    Snapshot target is not a snapshot table (missing "dbt_scd_id", "dbt_valid_from", "dbt_valid_to")

    > in macro materialization_snapshot_spark (macros/materializations/snapshot.sql)
    > called by snapshot snapshot_table (snapshots/snapshot_table.sql)

    After Fix
    ==========

    2021-08-22 22:10:59.192476 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: /* {"app": "dbt", "dbt_version": "0.19.1", "profile_name": "default", "target_name": "dev", "node_id": "snapshot.test_snapshot_model.snapshot_table"} */

        show databases

    2021-08-22 22:10:59.193652 (Thread-1): Opening a new connection, currently in state closed
    2021-08-22 22:10:59.557560 (Thread-1): Poll status: 2, query complete
    2021-08-22 22:10:59.558668 (Thread-1): SQL status: OK in 0.37 seconds
    2021-08-22 22:10:59.699644 (Thread-1): NotImplemented: add_begin_query
    2021-08-22 22:10:59.700834 (Thread-1): Using spark connection "snapshot.test_snapshot_model.snapshot_table".
    2021-08-22 22:10:59.701957 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: /* {"app": "dbt", "dbt_version": "0.19.1", "profile_name": "default", "target_name": "dev", "node_id": "snapshot.test_snapshot_model.snapshot_table"} */

        describe extended test_schema.snapshot_table

    2021-08-22 22:10:59.952625 (Thread-1): Poll status: 2, query complete
    2021-08-22 22:10:59.954015 (Thread-1): SQL status: OK in 0.25 seconds
    2021-08-22 22:11:00.083370 (Thread-1): Using spark connection "snapshot.test_snapshot_model.snapshot_table".

    2021-08-22 22:11:01.813533 (Thread-1): On snapshot.test_snapshot_model.snapshot_table: /* {"app": "dbt", "dbt_version": "0.19.1", "profile_name": "default", "target_name": "dev", "node_id": "snapshot.test_snapshot_model.snapshot_table"} */

        merge into test_schema.snapshot_table as DBT_INTERNAL_DEST
        using test_schema.snapshot_table__dbt_tmp as DBT_INTERNAL_SOURCE
        on DBT_INTERNAL_SOURCE.dbt_scd_id = DBT_INTERNAL_DEST.dbt_scd_id
        when matched
        and DBT_INTERNAL_DEST.dbt_valid_to is null
        and DBT_INTERNAL_SOURCE.dbt_change_type in ('update', 'delete')
            then update
            set dbt_valid_to = DBT_INTERNAL_SOURCE.dbt_valid_to

        when not matched
        and DBT_INTERNAL_SOURCE.dbt_change_type = 'insert'
            then insert *
        ;


    2021-08-22 22:11:06.893082 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:11.935455 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:16.977407 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:22.019852 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:27.101953 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:32.143668 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:37.185209 (Thread-1): Poll status: 1, sleeping
    2021-08-22 22:11:40.721562 (Thread-1): Poll status: 2, query complete
    2021-08-22 22:11:40.722843 (Thread-1): SQL status: OK in 38.91 seconds