Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Report sync and bugfixes #13

Merged
merged 128 commits into from
Feb 24, 2022
Merged

Report sync and bugfixes #13

merged 128 commits into from
Feb 24, 2022

Conversation

bryantgray
Copy link
Contributor

Description of change

  • Reports with one google ads resource sync properly
  • Added bookmarks and state to report sync
  • Added hashed PK for report sync and tests for hash
  • Fixed bugs with field exclusion
  • Updated field names to snake case
  • Made foreign keys automatic for core streams
  • Made date automatic for report streams

Manual QA steps

  • Running tap to target using real ad data

Risks

  • Pre-alpha.

Rollback steps

  • revert this branch

@bryantgray
Fix metrics field exclusion bug
008b8d2
@bryantgray
Enable report discovery and sync
47b41be
@bryantgray
Remove the prefix from fields names in the resource schema and json s…
b50de0a 
…chema
@bryantgray
Fix bug caused by shared references
cb95e45
@bryantgray
Use a tuple literal
8caa8db
@bryantgray
Add table-foreign-key-properties to stream metadata
54eae0f
@bryantgray
Add function to create nested JSON schema for core streams
fb6f5b0
@bryantgray
Add tests for create_nested_resource_schema
c2cf1be
@bryantgray
Refactor function to allow reuse with reports
e285226
@bryantgray
Fix tests after refactor
2532335
@bryantgray
Working sync for core streams
f2299f9
@bryantgray
Fix field exclusion and metadata bugs
e3ee4a1
@bryantgray
Add ReportStream class, update PK names
db030ed
@bryantgray
Remove flatten because schema is no longer flat
6a0a856
@bryantgray
Fix primary key bug
b485e51
@bryantgray
Require date as a segment
196d888
@bryantgray
Fix ads stream bugs
e549e28
@bryantgray
Add remapping for ads in report
2eed2f6
@bryantgray
WIP Incremental sync for report streams
e654e9f
@bryantgray
Make state global
b29583f
@bryantgray
Add hash of attributes and segments as PK for report streams
bba91ab
@bryantgray
Change reports to use the ReportStream class
a9e744a
@bryantgray
Change adgroupperformancereport to use report stream, don't write sta…
365a893 
…te as

often
@bryantgray
Make pylint happy
73cbe8f
@bryantgray
Change fields-to-sync to tap-google-ads.api-field-names
2ec3107 
Co-authored-by: leslie vandemark [email protected]
@bryantgray
Merge remote-tracking branch 'origin/main' into report-sync
2c2e9d7
@bryantgray
Fix record hash and add tests for record hash
eb3140e
@bryantgray
Make inclusion automatic on foreign keys
d4cd071
@bryantgray
Add docstring to create_resource_schema
5a4b937
@bryantgray
Change metadate fields_to_sync to tap-google-ads.api-field-names, don't
b9d0d6a 
look for api-field-names on record hash
dsprayberry and others added 19 commits February 22, 2022 16:14
@dsprayberry
Regroup functions for readability, rename sync_* to sync, set currently
a046de3 
syncing in core streams
@dsprayberry
Revert to conversion_window per discussed naming conventions
50f74fc
@dsprayberry
Clear currently_syncing after stream completion
d270c20
@dsprayberry
Fix bug TDL-17887, whitespace clean up
c9ca6d0
@dsprayberry
Fix import for create_nested_resource_schema
cc668d2
@dsprayberry
Remove tests for deleted functions
f697857
@dsprayberry
Update import for generate_hash
4dca226
@dsprayberry
Add get_query_date, add unit tests for get_query_date
0addaa8
@dsprayberry
Add unit tests to circle runs
4a62143
@dsprayberry
Finish refactor - call the new function name
20d08d1
@dsprayberry
Fix imports in unit test
d48cc7f
@dsprayberry
Fix bug in refactor: update arg name
3ca8a46
@dsprayberry
Remove default value in get_bookmark
28d07fe
@dsprayberry
Pass in a datetime string to get_query_date
171c07d
@dsprayberry
Return datetime objects from get_query_date
f27ba4b
@dsprayberry
Clean up TODO
91f0402
@dsprayberry
Fix unit tests
6330e58
@bhtowles
Bookmark updates, test more streams, remove lookback from start_date
f398111
@bhtowles
minor cleanup bookmarks test
e6a9a20
kspeer825
kspeer825 reviewed Feb 23, 2022
View reviewed changes
Copy link
Contributor

@kspeer825 kspeer825 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lett comments on tests for QA tracking.

@dsprayberry dsprayberry merged commit 23a045d into main Feb 24, 2022
@dsprayberry dsprayberry deleted the report-sync branch February 24, 2022 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dsprayberry dsprayberry dsprayberry left review comments

@kspeer825 kspeer825 kspeer825 left review comments

@cosimon cosimon cosimon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@bryantgray @cosimon @dsprayberry @kspeer825 @luandy64 @bhtowles