TDL-17511-Add missing test case, TDL-17522-Fix typos in the fields #5
Conversation
umangagrawal-crest
requested review from
dbshah1212,
karanpanchal-crest and
savan-chovatiya
| @@ -158,7 +158,7 @@ | |||
| ], | |||
| "format": "date-time" | |||
| }, | |||
| "largest_purchase item_price": { | |||
| "largest_purchase_item_price": { | |||
There was a problem hiding this comment.
The data was not collected for this field largest_purchase_item_price due to a typo issue. So corrected the field name.
| @@ -180,9 +180,9 @@ def sync(self, | |||
| with metrics.record_counter(self.tap_stream_id) as counter: | |||
| for record in self.get_records(bookmark_datetime): | |||
| self.date_records_to_datetime(record) | |||
| transform_keys_to_snake_case(record) | |||
There was a problem hiding this comment.
Transforming record from camel_case to snake_case before retrieving a value of replication key from the record as replication_key is defined in snake_case in the schema file
tap-sailthru/tap_sailthru/schemas/purchase_log.json
Lines 6 to 13 in 249d3ae
There was a problem hiding this comment.
Can you please attach the card related to this bug? To explain this issue in detail.
There was a problem hiding this comment.
Here is the bug card: https://jira.talendforge.org/browse/TDL-17522. We have made this change for the "date" field, to maintain consistency in the record.
| replication_key = 'Date' | ||
| valid_replication_keys = ['Date'] | ||
| replication_key = 'date' | ||
| valid_replication_keys = ['date'] |
There was a problem hiding this comment.
As mentioned above, syncing with the schema file.
tests/base.py
Outdated
| @@ -1,6 +1,11 @@ | |||
| import os | |||
| import unittest | |||
|
|
|||
| import singer | |||
There was a problem hiding this comment.
This can be removed, it's not used anywhere.
tests/test_bookmark.py
Outdated
| new_states = first_sync_bookmarks | ||
| menagerie.set_state(conn_id, new_states) |
There was a problem hiding this comment.
As written this does nothing. We get the state and put the same state back in menagerie. Ideally we should be moving the bookmarked values backward (decreasing the value) from those saved in state from sync 1. This way we see more data replicated in sync 2 than just the records which were bookmarked on from sync 1.
There was a problem hiding this comment.
@kspeer825 Updated the state between the sync and set a date smaller than bookmark value.
* TDL-17511-Add missing test case, TDL-17522-Fix typos in the fields (#5) * Added all the integration test cases * Added one assertion in discovery test * used pytest with coverage * removed un-used line from config.yml file * reverted the change * updated config.yml file for unittest coverage * removed user_agent from base test file * skip blast_query from start date and bookmark tests * removed unsed imports and updated state bookmark in bookmark test * removed setuptools * resolved bookmark test failure * resolved bookmark test failure * include blast_query stream in test cases Co-authored-by: harshpatel4_crest <[email protected]> * TDL-17521: Fix date_windowing for the purchase_log stream (#9) * Added all the integration test cases * updated date window for purchase log stream * run purchase log stream in start date test * Added one assertion in discovery test * used pytest with coverage * removed un-used line from config.yml file * reverted the change * updated config.yml file for unittest coverage * removed user_agent from base test file * skip blast_query from start date and bookmark tests Co-authored-by: Umang Agrawal <[email protected]> * TDL-17514: Implement request timeout (#6) * Added request timeout * Resolved review comments * run CCi build * updated config,yml file for unittest reports * removed user_agent and imported tap tester libs in base test file Co-authored-by: harshpatel4_crest <[email protected]> * TDL-17515: Verify credentials in discover mode (#7) * Added credentials check in discover mode * updated unittest and run CCi * removed user_agent from base file test * imported runner in base test file * imported menagerie in base test file Co-authored-by: harshpatel4_crest <[email protected]> * TDL-17516: Add custom exception handling (#8) * added exception handling * updated 404 error message * added 500 error class and comments * run CCi build and updated unittest assertion * updated config.yml file * updated config.yml file * removed user_agent from base test file * imported tap tester libs in base * TDL-17520: Primary Key is not unique for the purchase_log stream (#10) * updated primary key for purchase_log stream * removed user_agent from base test file * imported tap_tester in base test file * added channel in the primary key for purchase log * resolve pylint error * resolve unittest failure * skip 403 and 99 error, updated automatic test case for purchase_log stream Co-authored-by: Umang Agrawal <[email protected]> Co-authored-by: Umang Agrawal <[email protected]> Co-authored-by: savan-chovatiya <[email protected]>
somethingmorerelevant
deleted the
TDL-17511-Add-missing-tap-teseter-test
branch
Description of change
TDL-17511 Add missing tap tester tests
Added following below integration tests:
NOTE: Pagination test cases are not written as no stream is using pagination.
TDL-17522 FIx typos in the fields
largest_purchase item_pricein the schema file of blast_save_list stream.NOTE:
Manual QA steps
Risks
Rollback steps