Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add suffix to duplicate passthrough fields #15

Merged
merged 30 commits into from
Nov 13, 2024
Merged

Conversation

fivetran-jamie
Copy link
Contributor

@fivetran-jamie fivetran-jamie commented Nov 12, 2024

PR Overview

This PR will address the following Issue/Feature:
Though conversions data lives in the new source report tables we included, there are some legacy fields in the pre-existing reports that users may have included and aliased as conversions. This will ensure we avoid duplicate column errors by applying a _c suffix to any conversions fields previously included by the user via pass through column variables.

This PR will result in the following new package version:

Could be v0.3.1, as this is a fix to the already breaking v0.3.0 release. Though perhaps there's no harm in making it v0.4.0?

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

  • dbt run –full-refresh && dbt test
  • dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

  • The appropriate issue has been linked, tagged, and properly assigned
  • All necessary documentation and version upgrades have been applied
  • docs were regenerated (unless this PR does not include any code or yml updates)
  • BuildKite integration tests are passing
  • Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

Validation tests passing
image

If you had to summarize this PR in an emoji, which would it be?

🛃

@fivetran-jamie fivetran-jamie marked this pull request as ready for review November 12, 2024 23:05
Copy link
Collaborator

@fivetran-joemarkiewicz fivetran-joemarkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -20,6 +20,25 @@ vars:

reddit_ads__ad_conversions_passthrough_metrics:
- name: 'view_through_conversion_attribution_window_week'
reddit_ads__account_passthrough_metrics:
Copy link

@fivetran-avinash fivetran-avinash Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we want to keep these new variable configurations for the default dbt_project.yml Buildkite runs, or should we comment them out?

If we want to keep them, we'd probably want to add these to the source file as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah just wanted to make sure this passes on all warehouses.

What do you mean by source file?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bleh, dbt_reddit_ads_source, just so the dbt_project.yml files are matching.

Something to do later though!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah gotcha, yah later haha

reddit_ads__ad_passthrough_metrics:
- name: conversions
- name: app_install_metrics_install
alias: installs

# For validation testing
reddit_ads__conversion_event_types:
Copy link

@fivetran-avinash fivetran-avinash Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should these be commented out before merging since it's just for validation tests, or is there no harm, no foul here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no harm in leaving them IMO, more testing

- Ensures the addition of conversion metrics in [v0.3.0](https://github.com/fivetran/dbt_reddit_ads/blob/main/CHANGELOG.md#dbt_reddit_ads-v030) is truly backwards compatible and will avoid duplicate column errors regardless of any pre-existing configurations.
- Specifically, if you were previously utilizing [passthrough column](https://github.com/fivetran/dbt_reddit_ads?tab=readme-ov-file#passing-through-additional-metrics) variables to include fields called `conversions`, `view_through_conversions`, `total_value`, or `total_items`, the package's version of these fields will take precedence. Your fields will be included as well, but suffixed with a `_c`.

## Under the Hood
Copy link

@fivetran-avinash fivetran-avinash Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should call out the modifications to the consistency tests as well regarding the new measures we are now testing/comparing between dev and prod.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding

@@ -126,6 +126,14 @@ vars:
- name: "new_custom_field"
alias: "custom_field"
- name: "a_second_field"
reddit_ads__account_conversions_passthrough_metrics:
Copy link

@fivetran-avinash fivetran-avinash Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably provide an example of how the alias would be applied for a few of these variables.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added!

Copy link

@fivetran-avinash fivetran-avinash Nov 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-jamie I'm wondering if we should call out the backwards-compatibility functionality of conversions fields, or whether it should live in the DECISIONLOG. This update might not be quite intuitive to a customer unless they really go hunting for the macro. What are your thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good callout -- I think the decision log makes the most sense

Copy link

@fivetran-avinash fivetran-avinash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-jamie left a few comments!

Copy link

@fivetran-avinash fivetran-avinash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fivetran-jamie Approved!

Two final suggestions: Can you add a Documentation Update section to the CHANGELOG and link to the DECISIONLOG there? (Optional, but you could also link to the README as well).

@fivetran-jamie fivetran-jamie merged commit 2363c9c into main Nov 13, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants