Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate away from Singer-based connectors #3582

Closed
sherifnada opened this issue May 25, 2021 · 2 comments
Closed

Migrate away from Singer-based connectors #3582

sherifnada opened this issue May 25, 2021 · 2 comments

Comments

@sherifnada
Copy link
Contributor

sherifnada commented May 25, 2021

Tell us about the problem you're trying to solve

Airbyte started off as a Singer-based tool. We've since migrated to an Airbyte protocol that's compatible with Singer. As a result, we've built a number of connectors based on Singer taps. This presents a few points of friction:

  1. We need to fork the singer tap into a separate repo which presents all sorts of headaches:
  2. Need to grant access permissions separately
  3. We need to use git-based pip installs like this one
  4. The iteration process is more friction-ful since you need to create a change in one repo, push it, then use branch-based pip installs like in Singer Postgres --> Postgres replication demo #2 (e.g: `pip install
  5. We need to tag commits which is not part of the standard workflow for connectors in github.com/airbytehq/airbyte
  6. To onboard someone into the connectors codebase they need to know both the Singer protocol and ecosystem as well as the Airbyte tooling
  7. All these painpoints combined make it harder to come in as a community contributor to contribute to a Singer-based connector
  8. Singer taps vary in quality and support levels. Many of them receive no support from stitch.
  9. It's harder to use Airbyte native auto generators for generating unit tests
  10. Singer tap code is not uniform due to lack of abstractions/frameworks provided by stitch. This makes it more difficult to break into singer code. Airbyte-CDK based connectors should be much faster to understand for someone familiar with the patterns.

Describe the solution you’d like

I would like to migrate all of our Singer connectors to be Airbyte-native ones so we avoid the issues above.

Here are all the Singer-based connectors we currently offer:

  • Appsflyer
  • Appstore
  • Braintree
  • Github
  • Gitlab
  • Google Adwords
  • Google Analytics
  • Google Search Console
  • Intercom
  • Marketo
  • Mixpanel
  • Quickbooks
  • Salesforce
  • Shopify
  • Slack
  • Twilio
  • Zendesk Suppot
  • Zoom

We should create tickets for these and one by one start migrating away from Singer

┆Issue is synchronized with this Asana task by Unito

@marcosmarxm
Copy link
Member

@sherifnada wdyt creating specific issues New Native Source: Marketo for example?
There are new information the community brings and could be nice add into the issue. This way the contributor who create the new version have knowledge about some actual limitation.

@sherifnada
Copy link
Contributor Author

@marcosmarxm SGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants