Implement a common config and validator #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maksenius
approved these changes
Sep 27, 2022
oykmnk
pushed a commit
that referenced
this pull request
Oct 25, 2022
lovromazgon
added a commit
that referenced
this pull request
Aug 10, 2023
* implement common config * provide source config * add one more test case for the source config * implement hubspot http client * rename test functions * update comment * update dependencies * enable check shadowing * capture resp body if unexpected status code is occured * add `gte` and `lte` validations * implement snapshot iterator * provide the `MaxRetries` config field * add comment about hubspot's id * update `poll`'s comment * provide a consts for id and createdAt fields * move iterator's errors to a separate file * regenerate mocks * add hubspot resource validation * implement destination * pass a resource to the writer * update the objectIdPlaceholder * add teardown log * update destination comments * Implement CDC iterator (#8) * implement cdc iterator * limit items in cdc iterator * refactor multiline function arguments * add comment to the cdc createdAt condition * update comment * check for nil iterator on teardown (#9) * Implement Source integration tests (#10) * implement source integration tests * rename integration tests * update function signature * remove redundant `waitTestContacts` function * cancel context and teardown with a new one * Implement Destination integration tests (#12) * implement source integration tests * rename integration tests * update function signature * update source integration test * implement destination integration tests * add comment to the test target * remove redundant `waitTestContacts` function * cancel context and teardown with a new one * destination integration tests add cancel * add source config validation * Implement deletion handling in CDC iterator (#13) * deletion handling in cdc * linter fix * pr fixes #1 * pr fixes #2 * comment added * dot fix * replace sending to channel with return * Update CDC and fix `bufferSize` and HubSpot limit functionality (#14) * remove resources that don't support cdc * update snapshot iterator - fix the incorrect behavior with the bufferSize and limit * simplify Snapshot's HasNext method * fix typo * move Snapshot's Stop method above the private methods * add sorting for timestamp-based resources * update sorting logic and switching to cdc mode * store initial snapshot timestamp in the position * reformat arguments * set the snapshot's initial timestamp to the cdc timestamp when switching * remove redundant ConvertToCDCPosition function * rename `do request` to `execute request` * remove unsupported resources from the destination * Add Source documentation (#15) * source documentation, first run * add empty resources.md * fmt README * fix typo * update configuration options documentation * `resource` -> `resources` * format source config table * add resource and source operations they support * Add Destination documentation (#16) * add destination operations resources support * add destination docs * Implement acceptance tests (#17) * use a different context for deletes in destination tests * implement acceptance tests * update `WriteToSource` method * acceptance tests minor fixes * wait for test data to be present, in integration tests * Add `extraProperties` source config field (#19) * provide a new source config field `extraProperties` This field allows to specify a list of HubSpot resource properties to include in addition to the default. If any of the specified properties are not present on the requested HubSpot resource, they will be ignored. Only CRM resources support this. * add `extraProperties` format to the docs * fix test after merge * update resources list, rename `insert` to `create` * Add new source config field, `snapshot` (#23) * Data duplication fix (#25) * Make `Position.ItemID` a string (#26) * fix itemID check * Update connector - update dependencies - update go version - fix golangci-lint config - fix the way cleanup is done in destination integration tests * refactor tests * fix linter --------- Co-authored-by: BohdanMyronchuk <[email protected]> Co-authored-by: aleksandr.yakimenko <[email protected]> Co-authored-by: Lovro Mažgon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR seeks to implement a common config and validator.
Quick checks: