Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

example of tap interface (with example implementation of postgres) #6

Closed

Conversation

cgardens
Copy link
Contributor

@cgardens cgardens commented Aug 3, 2020

This PR attempts to describe the interface we would provide to implement taps against to get feedback on whether the interface makes sense.

pg_integration/pgConfig.json describes all the types needed to implement a postgres tap. types prefaced with "standard" are types that will be used in ALL taps. types prefaced with "postgres".

pg_integration/src/gen/postgresTap.ts are the typescript types generated from pg_integration/pgConfig.json. used https://app.quicktype.io/ to generate the code from the json-schema formatted json.

pg_integration/src/app.ts describes the generic interface that will be used for each tap. then it also shows what that interface looks like when implemented for postgres. also added an example of what it might look like to have a generic singer tap so that the singer internals don't need to be implemented multiple times. in the case where we are trying to use an existing singer tap we just need to implement converters to and from singer and our types.

@cgardens cgardens changed the title example of tap interface example of tap interface (with example implementation of postgres) Aug 3, 2020
connectionConfiguration: PostgresConnectionConfiguration,
syncConfiguration: PostgresSyncConfiguration,
standardSyncConfiguration: StandardSyncConfiguration,
state: any
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a weakness in the abstraction. essentially we want to keep a blob that is output from singer (the state.json). store it somewhere and then pass it in on the next run of sync. it is unclear to me if we should handle this. do we just treat it like a blob? do we just define the json schema for it? not sure yet. probably should just look at the postgres singer tap and define the schema on our side based on what's there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one reason i'm not loving this is that i have at least run into singer catalogs that are incredibly difficult (if not impossible) to declare in json schema. this is because the catalogs are whacky. the singer postgres tap catalog as an example: https://github.com/datalineio/conduit/blob/078504ff9ccd20f84cd0b17eac196bd9d38f323c/psql2psql/local_tap_catalog.json. so we just need to keep our fingers crossed that we don't encounter any state files that are too whacky.

@cgardens cgardens marked this pull request as draft August 11, 2020 22:51
@cgardens
Copy link
Contributor Author

The contents of this PR have been adapted into our docs and java implementation.

@cgardens cgardens closed this Aug 27, 2020
mohamagdy added a commit to mohamagdy/airbyte that referenced this pull request May 1, 2022
ahmed-buksh pushed a commit to ahmed-buksh/airbyte that referenced this pull request May 25, 2022
@sbjorn sbjorn mentioned this pull request Aug 29, 2022
18 tasks
matter-q added a commit that referenced this pull request Sep 19, 2022
matter-q pushed a commit that referenced this pull request Sep 22, 2022
* Replaced sidebar menu with headlessui

* Removed unnecessary tag li

* Removed unnecessary Sidebar Popout

* Changed export type

* Added keyboard support

* Removed styled-components

* Removed commented code

* Disabled ESLint rule css-modules/no-undef-class

* Review fixes

* Fixed shorthand property

* Review fixes

* Review fixes #2

* Review fixes #3

* Review fixes #4

* Review fixes #5

* Review fixes #6

* Review fixes #7

* Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss

Co-authored-by: Vladimir <[email protected]>

* Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss

Co-authored-by: Vladimir <[email protected]>

Co-authored-by: Vladimir <[email protected]>
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
* Replaced sidebar menu with headlessui

* Removed unnecessary tag li

* Removed unnecessary Sidebar Popout

* Changed export type

* Added keyboard support

* Removed styled-components

* Removed commented code

* Disabled ESLint rule css-modules/no-undef-class

* Review fixes

* Fixed shorthand property

* Review fixes

* Review fixes airbytehq#2

* Review fixes airbytehq#3

* Review fixes airbytehq#4

* Review fixes airbytehq#5

* Review fixes airbytehq#6

* Review fixes airbytehq#7

* Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss

Co-authored-by: Vladimir <[email protected]>

* Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss

Co-authored-by: Vladimir <[email protected]>

Co-authored-by: Vladimir <[email protected]>
@swyxio swyxio deleted the charles/sample_pg_integration2 branch October 11, 2022 15:53
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
* Replaced sidebar menu with headlessui

* Removed unnecessary tag li

* Removed unnecessary Sidebar Popout

* Changed export type

* Added keyboard support

* Removed styled-components

* Removed commented code

* Disabled ESLint rule css-modules/no-undef-class

* Review fixes

* Fixed shorthand property

* Review fixes

* Review fixes airbytehq#2

* Review fixes airbytehq#3

* Review fixes airbytehq#4

* Review fixes airbytehq#5

* Review fixes airbytehq#6

* Review fixes airbytehq#7

* Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss

Co-authored-by: Vladimir <[email protected]>

* Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss

Co-authored-by: Vladimir <[email protected]>

Co-authored-by: Vladimir <[email protected]>
@avirajsingh7 avirajsingh7 mentioned this pull request Aug 3, 2023
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant