🎉 Source Iterable - added new events streams #16067

midavadim · 2022-08-29T08:20:04Z

What

#13504 - Update Iterable connector to support incremental events

'events' API does not support the ability to query on specific date range:
https://api.iterable.com/api/docs#export_exportUserEvents
Export all events in JSON format for a user-specified by email or userId. One event per line.
so it is not possible to implement incremental sync for 'events' streams based on exportUserEvents API.

How

So exportUserEvents and exportDataJson returns the same data, explanation EXPORT API:
https://api.iterable.com/api/docs#export_exportUserEvents:
export all events (all types) by user
https://api.iterable.com/api/docs#export_exportDataJson:
export all events by event_type, support incremental sync based on date range

So since exportUserEvents does not allow incremental sync, we can implement separate incremental streams for all missing event types. thus such implementation covers customer request.

Important: our current iterable account does not have any date for new streams.

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-08-29T08:20:08Z

All committers have signed the CLA.

midavadim · 2022-08-29T08:20:26Z

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2947011184
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2947011184
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%
Name                                  Stmts   Miss  Cover
---------------------------------------------------------
source_iterable/source.py                18      0   100%
source_iterable/slice_generators.py      73      0   100%
source_iterable/__init__.py               2      0   100%
source_iterable/iterable_streams.py     105      7    93%
source_iterable/api.py                  214     25    88%
---------------------------------------------------------
TOTAL                                   412     32    92%

Build Passed

Test summary info:

All Passed

lazebnyi

LGTM

airbyte-integrations/connectors/source-iterable/unit_tests/test_export_adjustable_range.py

YowanR · 2022-08-30T00:40:54Z

Assigning this one to @alafanechere since @Phlair is OOO.

alafanechere

Thanks for this addition. I made multiple suggestions.
What would it take to fill our sandbox account with test data on these new streams to ensure our SAT runs on this PR?
If you can't feed the sandbox account with test data could you please declare the empty streams in the acceptance-test-config.yml?

In addition could you please bump the connector version and update the changelog?

airbyte-integrations/connectors/source-iterable/source_iterable/api.py

airbyte-integrations/connectors/source-iterable/unit_tests/test_api.py

airbyte-integrations/connectors/source-iterable/unit_tests/test_source.py

brianjlai

Behaviorally things look, good but as mentioned before, I think we should restructure api.py and iterable_streams.py to adhere to existing connector conventions. Once we do that, pending retesting, we should be all set.

docs/integrations/sources/iterable.md

airbyte-integrations/connectors/source-iterable/source_iterable/api.py

midavadim · 2022-09-02T21:46:27Z

/test connector=connectors/source-iterable

brianjlai

changes look good, thanks for restructuring the streams. it might still be worth looking into what data we could populate to adequately test the streams in SAT, but that can be a follow up.

lazebnyi · 2022-09-05T05:34:01Z

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2990976111
❌ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2990976111
🐛 https://gradle.com/s/olius5muczqzy

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 ERROR unit_tests/test_exports_stream.py
	 ERROR unit_tests/test_streams.py
	 !!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!
	 �[31m============================== �[31m�[1m2 errors�[0m�[31m in 1.39s�[0m�[31m ===============================�[0m

lazebnyi · 2022-09-05T06:12:50Z

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2991108437
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2991108437
Python tests coverage:

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
source_iterable/source.py                18      0   100%
source_iterable/__init__.py               2      0   100%
source_iterable/slice_generators.py      73      5    93%
source_iterable/streams.py              314     35    89%
---------------------------------------------------------
TOTAL                                   407     40    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

…ts' into midavadim/14147-iterable-unittests # Conflicts: # airbyte-integrations/connectors/source-iterable/unit_tests/test_streams.py

midavadim · 2022-09-05T14:59:21Z

/publish connector=connectors/source-iterable

🕑 Publishing the following connectors:
connectors/source-iterable
https://github.com/airbytehq/airbyte/actions/runs/2994251395

Connector	Did it publish?	Were definitions generated?
connectors/source-iterable	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

midavadim · 2022-09-05T15:24:23Z

/publish connector=connectors/source-iterable

🕑 Publishing the following connectors:
connectors/source-iterable
https://github.com/airbytehq/airbyte/actions/runs/2994420449

Connector	Did it publish?	Were definitions generated?
connectors/source-iterable	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

lazebnyi · 2022-09-05T17:57:07Z

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2995234333
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/2995234333
Python tests coverage:

Name                                  Stmts   Miss  Cover
---------------------------------------------------------
source_iterable/source.py                18      0   100%
source_iterable/__init__.py               2      0   100%
source_iterable/slice_generators.py      73      5    93%
source_iterable/streams.py              314     35    89%
---------------------------------------------------------
TOTAL                                   407     40    90%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       121     25    79%   21-23, 29-31, 36-43, 48-61, 208-216
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       110     48    56%   23-26, 32, 36, 39-64, 67-69, 72-74, 77-79, 82-84, 87-89, 92-110, 144-146
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1322    463    65%

Build Passed

Test summary info:

All Passed

midavadim · 2022-09-05T19:34:35Z

/publish connector=connectors/source-iterable

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

lazebnyi · 2022-09-05T19:36:24Z

/publish connector=connectors/source-iterable run-tests=false

🕑 Publishing the following connectors:
connectors/source-iterable
https://github.com/airbytehq/airbyte/actions/runs/2995668936

Connector	Did it publish?	Were definitions generated?
connectors/source-iterable	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…ts' into midavadim/14147-iterable-unittests

* increased unit test coverage * added additional events streams * updated tests * bumped connector version, update changelog * fixed indentations * api.py and iterable_streams.py merged into one stream.py file * Fix unit tests * updated release stage * fixed import * Updated version in seed * auto-bump connector version [ci skip] * updated source_specs.yaml Co-authored-by: Serhii Lazebnyi <[email protected]> Co-authored-by: Serhii Lazebnyi <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

midavadim added 2 commits August 29, 2022 01:17

increased unit test coverage

da39c65

added additional events streams

7be155a

github-actions bot added the area/connectors Connector related issues label Aug 29, 2022

octavia-squidington-iii added the connectors/source/iterable label Aug 29, 2022

This was linked to issues Aug 29, 2022

[EPIC] Source Iterable to Beta #14147

Closed

Update Iterable connector to support incremental events #13504

Closed

midavadim requested review from lazebnyi and bazarnov August 29, 2022 08:37

bazarnov approved these changes Aug 29, 2022

View reviewed changes

lazebnyi approved these changes Aug 29, 2022

View reviewed changes

airbyte-integrations/connectors/source-iterable/unit_tests/test_export_adjustable_range.py Show resolved Hide resolved

midavadim requested a review from Phlair August 29, 2022 08:59

lazebnyi self-assigned this Aug 29, 2022

lazebnyi removed a link to an issue Aug 29, 2022

[EPIC] Source Iterable to Beta #14147

Closed

10 tasks

YowanR requested a review from alafanechere August 30, 2022 00:40

alafanechere requested changes Sep 1, 2022

View reviewed changes

lazebnyi removed the request for review from Phlair September 2, 2022 12:57

updated tests

4d7e710

midavadim requested a review from alafanechere September 2, 2022 14:35

bazarnov approved these changes Sep 2, 2022

View reviewed changes

bumped connector version, update changelog

5f924e0

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 2, 2022

brianjlai reviewed Sep 2, 2022

View reviewed changes

docs/integrations/sources/iterable.md Show resolved Hide resolved

airbyte-integrations/connectors/source-iterable/source_iterable/api.py Outdated Show resolved Hide resolved

midavadim added 2 commits September 2, 2022 23:53

fixed indentations

372289d

api.py and iterable_streams.py merged into one stream.py file

5d725cb

midavadim requested a review from brianjlai September 2, 2022 21:46

brianjlai approved these changes Sep 3, 2022

View reviewed changes

lazebnyi requested review from alafanechere and removed request for alafanechere September 5, 2022 05:34

Fix unit tests

a3a5044

midavadim added 2 commits September 5, 2022 17:54

updated release stage

26be730

Merge remote-tracking branch 'origin/midavadim/14147-iterable-unittes…

5c3bbc6

…ts' into midavadim/14147-iterable-unittests # Conflicts: # airbyte-integrations/connectors/source-iterable/unit_tests/test_streams.py

midavadim temporarily deployed to more-secrets September 5, 2022 14:57 Inactive

fixed import

f98f699

midavadim temporarily deployed to more-secrets September 5, 2022 15:21 Inactive

lazebnyi and others added 2 commits September 5, 2022 20:59

Merge branch 'master' into midavadim/14147-iterable-unittests

dd33d75

Updated version in seed

7fa7688

lazebnyi temporarily deployed to more-secrets September 5, 2022 19:01 Inactive

lazebnyi temporarily deployed to more-secrets September 5, 2022 19:04 Inactive

octavia-squidington-iii and others added 3 commits September 5, 2022 19:53

auto-bump connector version [ci skip]

c32c153

updated source_specs.yaml

fdda85e

Merge remote-tracking branch 'origin/midavadim/14147-iterable-unittes…

057525c

…ts' into midavadim/14147-iterable-unittests

midavadim temporarily deployed to more-secrets September 5, 2022 19:59 Inactive

midavadim merged commit 7ae67d1 into master Sep 5, 2022

midavadim deleted the midavadim/14147-iterable-unittests branch September 5, 2022 20:00

octavia-squidington-iii mentioned this pull request Sep 12, 2022

Bump Airbyte version from 0.40.4 to 0.40.5 #16610

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Iterable - added new events streams #16067

🎉 Source Iterable - added new events streams #16067

midavadim commented Aug 29, 2022 •

edited

Loading

CLAassistant commented Aug 29, 2022 •

edited

Loading

midavadim commented Aug 29, 2022 •

edited by github-actions bot

Loading

lazebnyi left a comment

YowanR commented Aug 30, 2022

alafanechere left a comment

brianjlai left a comment

midavadim commented Sep 2, 2022

brianjlai left a comment

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

midavadim commented Sep 5, 2022 •

edited by github-actions bot

Loading

midavadim commented Sep 5, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

midavadim commented Sep 5, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

🎉 Source Iterable - added new events streams #16067

🎉 Source Iterable - added new events streams #16067

Conversation

midavadim commented Aug 29, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Aug 29, 2022 • edited Loading

midavadim commented Aug 29, 2022 • edited by github-actions bot Loading

Build Passed

lazebnyi left a comment

Choose a reason for hiding this comment

YowanR commented Aug 30, 2022

alafanechere left a comment

Choose a reason for hiding this comment

brianjlai left a comment

Choose a reason for hiding this comment

midavadim commented Sep 2, 2022

brianjlai left a comment

Choose a reason for hiding this comment

lazebnyi commented Sep 5, 2022 • edited by github-actions bot Loading

Build Failed

lazebnyi commented Sep 5, 2022 • edited by github-actions bot Loading

Build Passed

midavadim commented Sep 5, 2022 • edited by github-actions bot Loading

midavadim commented Sep 5, 2022 • edited by github-actions bot Loading

lazebnyi commented Sep 5, 2022 • edited by github-actions bot Loading

Build Passed

midavadim commented Sep 5, 2022 • edited by github-actions bot Loading

lazebnyi commented Sep 5, 2022 • edited by github-actions bot Loading

midavadim commented Aug 29, 2022 •

edited

Loading

CLAassistant commented Aug 29, 2022 •

edited

Loading

midavadim commented Aug 29, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

midavadim commented Sep 5, 2022 •

edited by github-actions bot

Loading

midavadim commented Sep 5, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading

midavadim commented Sep 5, 2022 •

edited by github-actions bot

Loading

lazebnyi commented Sep 5, 2022 •

edited by github-actions bot

Loading