Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add DATA WAREHOUSE to integrations object in shopify pixel source #3980

Open
wants to merge 5 commits into
base: chore.redis.support.shopify.pixel
Choose a base branch
from

Conversation

yashasvibajpai
Copy link
Member

@yashasvibajpai yashasvibajpai commented Jan 13, 2025

What are the changes introduced in this PR?

We will need to append this integrations object to every incoming event coming through the Shopify source. The path should point to the Shopify Details object inside the context object.

We send some events from Shopify to Warehouse destinations. Now, within the sent event’s context, there is a field called shopifyDetails which actually serves as a dump for the untransformed event received from Shopify by our source transformer. Now, if this shopifyDetails is sent as is, because of its large size and large number of fields within that object, it would result in large number of columns getting created within the warehouse.
so for this we want to leverage JSON paths.

integrations: {
                    SHOPIFY: true,
                    DATA_WAREHOUSE: {
                      options: {
                        jsonPaths: [`${event type}.context.shopifyDetails`],
                      },
                    },
                  },

For 4 providers as of now (rs, postgres, snowflake and bq), we wanted to support a new value within the integrations object. DATA_WAREHOUSE that will allow us to propagate shopifyDetails to be a jsonpath from the source event.

What is the related Linear task?

Resolves INT-3098 INT-3097

Please explain the objectives of your changes below

Put down any required details on the broader aspect of your changes. If there are any dependent changes, mandatorily mention them here

Any changes to existing capabilities/behaviour, mention the reason & what are the changes ?

N/A

Any new dependencies introduced with this change?

N/A

Any new generic utility introduced or modified. Please explain the changes.

N/A

Any technical or performance related pointers to consider with the change?

N/A

@coderabbitai review


Developer checklist

  • My code follows the style guidelines of this project

  • No breaking changes are being introduced.

  • All related docs linked with the PR?

  • All changes manually tested?

  • Any documentation changes needed with this change?

  • Is the PR limited to 10 file changes?

  • Is the PR limited to one linear task?

  • Are relevant unit and component test-cases added in new readability format?

Reviewer checklist

  • Is the type of change in the PR title appropriate as per the changes?

  • Verified that there are no credentials or confidential data exposed with the changes.

@devops-github-rudderstack
Copy link
Contributor

Allure Test reports for this run are available at:

Copy link

codecov bot commented Jan 13, 2025

Codecov Report

Attention: Patch coverage is 77.77778% with 4 lines in your changes missing coverage. Please review.

Project coverage is 90.73%. Comparing base (0fbe91a) to head (bd3e8e0).
Report is 1 commits behind head on chore.redis.support.shopify.pixel.

Files with missing lines Patch % Lines
src/warehouse/index.js 75.00% 4 Missing ⚠️
Additional details and impacted files
@@                        Coverage Diff                         @@
##           chore.redis.support.shopify.pixel    #3980   +/-   ##
==================================================================
  Coverage                              90.73%   90.73%           
==================================================================
  Files                                    623      623           
  Lines                                  32556    32570   +14     
  Branches                                7726     7739   +13     
==================================================================
+ Hits                                   29541    29554   +13     
+ Misses                                  2787     2758   -29     
- Partials                                 228      258   +30     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yashasvibajpai yashasvibajpai requested a review from a team as a code owner January 13, 2025 22:24
@devops-github-rudderstack
Copy link
Contributor

Allure Test reports for this run are available at:

@yashasvibajpai yashasvibajpai changed the base branch from develop to chore.redis.support.shopify.pixel January 14, 2025 04:27
@devops-github-rudderstack
Copy link
Contributor

Allure Test reports for this run are available at:

@yashasvibajpai yashasvibajpai requested a review from a team as a code owner January 15, 2025 19:38
@devops-github-rudderstack
Copy link
Contributor

Allure Test reports for this run are available at:

@devops-github-rudderstack
Copy link
Contributor

Allure Test reports for this run are available at:

@yashasvibajpai
Copy link
Member Author

@coderabbitai review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants