🎉 New Source-connector for Zuora

[bazarnov]

What

Resolving the issue:
#2664 - New Source-connector for Zuora

How

Using CDK and custom methods implemented the connector logic for Zuora as a source for Airbyte.

Connector Features

As a source of data Airbyte Zuora Connector provides users with the following capabilities:

• support of all objects (streams) available in Zuora account
• support of custom objects manually added by user
• support of custom fields in both standard zuora objects and custom objects
• support both full-refresh and incremental refresh with overwrite, append, append-deduplicate methods

Code features:

• the list of streams are dynamically retrieved from Zuora account
• stream schemas and catalog are built dynamically, based on the list of available objects in Zuora account
  (including the custom objects & custom fields)
• stream classes are built dynamically based on the list of available streams in Zuora account
• cursor_field is set dynamically for each individual stream in catalog
• cursor_field is automatically switched between updateddate or createddate, if the default cursor_field updateddate is not available to use in a particular stream
• the access token is being refreshed automatically if it appears to be invalid or expired
• if the particular stream cannot be processed due to Zuora account Permissions or Subscription Plan is not covering some features, the stream will be skipped, and log message would be provided in the output without stopping the other streams to be processed (specifically for the cases where the list of objects contains the stream as available for fetch, but not allowed to be processed due to permissions or subscription plan)

Pre-merge Checklist

Expand the checklist which is relevant for this PR.

Connector checklist

• Issue acceptance criteria met
• PR name follows PR naming conventions
• Secrets are annotated with airbyte_secret in the connector's spec
• Credentials added to Github CI if needed and not already present. instructions for injecting secrets into CI.
• Unit & integration tests added as appropriate (and are passing)
  • Community members: please provide proof of this succeeding locally e.g: screenshot or copy-paste acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• /test connector=connectors/<name> command as documented here is passing.
  • Community members can skip this, Airbyters will run this for you.
• Code reviews completed
• Documentation updated
  • README.md
  • docs/SUMMARY.md if it's a new connector
  • Created or updated reference docs in docs/integrations/<source or destination>/<name>.
  • Changelog in the appropriate page in docs/integrations/.... See changelog example
  • docs/integrations/README.md contains a reference to the new connector
  • Build status added to build page
• Build is successful
• Connector version bumped like described here
• New Connector version released on Dockerhub by running the /publish command described here
• No major blockers
• PR merged into master branch
• Follow up tickets have been created
• Associated tickets have been closed & stakeholders notified

[bazarnov]

/test connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1081972519
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1081972519

[Phlair]

Thanks for including custom integration tests, appreciate that!

Actual sync logic seems solid, made a couple of minor comments here and there.

Main thing to discuss is that I found the inheritance structure a little bit unintuitive, since mostly everything is ultimately a child of HttpStream & ZuoraStream. This would, for example, imply that ZuoraDescribeObject is a ZuoraStream by hierarchical inheritance. I can however see your reasoning here, to inherit a bunch of useful Http methods. I'd be tempted to create Mixins or just separate non-stream inherited classes for the non-stream Zuora logic (ZuoraBase, ZuoraListObjects, ZuoraDescribeObject, ZuoraSubmitJob , ZuoraJobStatusCheck ) and then have ZuoraObjectsBase either inherit from these mixins or instantiate those objects and use the logic appropriately. Thing is when it comes to structure there are many ways to skin the cat and there isn't exactly a correct answer, so happy to discuss this more. Wdyt?

Couple of other things, missing some additions to files:

• airbyte-integrations/builds.md
• docs/SUMMARY.md
• docs/integrations/README.md

[bazarnov]

Couple of other things, missing some additions to files:

• airbyte-integrations/builds.md
• docs/SUMMARY.md
• docs/integrations/README.md

Those are just added, I'll push the updates along with updated acceptance-test

[bazarnov]

/test connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1082429546
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1082429546

[Phlair]

@sherifnada
This source requires hitting a few different endpoints before eventually hitting the final one which streams the data (this isn't an auth flow though). Current implementation here defines classes that ultimately all inherit from HttpStream to handle these pre-requests. This feels slightly off to me because these aren't technically streams but they inherit to gain all the request handling goodness (like backoffs) without redefining the logic.

Oleksandr and I discussed this earlier which sparked an idea to support this better in cdk. We could separate a RequestHandler() based on the logic in HttpStream, which HttpStream can then instantiate and use to retain same functionality. This separate RequestHandler() could then be utilised in cases like this source where we actually need varying requests (path, headers, params, response parsing) within a single HttpStream().

wdyt?

[sherifnada]

@Phlair that sounds interesting, is it basically splitting out the client functionality for hitting the API out of the stream? Would you be able to illustrate with an example here or in an issue?

[Phlair]

Approved.

I'll create an issue based on the above conversations and link here

[bazarnov]

Approved.

I'll create an issue based on the above conversations and link here

Sounds great, I'll fix the conflicts of this branch and test again + push the connector to docker, then merge this PR. Thank you @Phlair

[bazarnov]

/test connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1098272374
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1098272374

[bazarnov]

/publish connector=connectors/source-zuora

🕑 connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1098491962
✅ connectors/source-zuora https://github.com/airbytehq/airbyte/actions/runs/1098491962

[Phlair]

created issue around points above: https://github.com/airbytehq/airbyte-internal-issues/issues/193

[bazarnov]

Many thanks, @Phlair