-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
example of tap interface (with example implementation of postgres) #6
example of tap interface (with example implementation of postgres) #6
Conversation
connectionConfiguration: PostgresConnectionConfiguration, | ||
syncConfiguration: PostgresSyncConfiguration, | ||
standardSyncConfiguration: StandardSyncConfiguration, | ||
state: any |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a weakness in the abstraction. essentially we want to keep a blob that is output from singer (the state.json). store it somewhere and then pass it in on the next run of sync
. it is unclear to me if we should handle this. do we just treat it like a blob? do we just define the json schema for it? not sure yet. probably should just look at the postgres singer tap and define the schema on our side based on what's there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
one reason i'm not loving this is that i have at least run into singer catalogs that are incredibly difficult (if not impossible) to declare in json schema. this is because the catalogs are whacky. the singer postgres tap catalog as an example: https://github.com/datalineio/conduit/blob/078504ff9ccd20f84cd0b17eac196bd9d38f323c/psql2psql/local_tap_catalog.json. so we just need to keep our fingers crossed that we don't encounter any state files that are too whacky.
The contents of this PR have been adapted into our docs and java implementation. |
…-dbt Call `DotMap` with square brackets
remove question response stream
* Replaced sidebar menu with headlessui * Removed unnecessary tag li * Removed unnecessary Sidebar Popout * Changed export type * Added keyboard support * Removed styled-components * Removed commented code * Disabled ESLint rule css-modules/no-undef-class * Review fixes * Fixed shorthand property * Review fixes * Review fixes #2 * Review fixes #3 * Review fixes #4 * Review fixes #5 * Review fixes #6 * Review fixes #7 * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <[email protected]> * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <[email protected]> Co-authored-by: Vladimir <[email protected]>
* Replaced sidebar menu with headlessui * Removed unnecessary tag li * Removed unnecessary Sidebar Popout * Changed export type * Added keyboard support * Removed styled-components * Removed commented code * Disabled ESLint rule css-modules/no-undef-class * Review fixes * Fixed shorthand property * Review fixes * Review fixes airbytehq#2 * Review fixes airbytehq#3 * Review fixes airbytehq#4 * Review fixes airbytehq#5 * Review fixes airbytehq#6 * Review fixes airbytehq#7 * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <[email protected]> * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <[email protected]> Co-authored-by: Vladimir <[email protected]>
* Replaced sidebar menu with headlessui * Removed unnecessary tag li * Removed unnecessary Sidebar Popout * Changed export type * Added keyboard support * Removed styled-components * Removed commented code * Disabled ESLint rule css-modules/no-undef-class * Review fixes * Fixed shorthand property * Review fixes * Review fixes airbytehq#2 * Review fixes airbytehq#3 * Review fixes airbytehq#4 * Review fixes airbytehq#5 * Review fixes airbytehq#6 * Review fixes airbytehq#7 * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <[email protected]> * Update airbyte-webapp/src/packages/cloud/views/layout/SideBar/SideBar.module.scss Co-authored-by: Vladimir <[email protected]> Co-authored-by: Vladimir <[email protected]>
This PR attempts to describe the interface we would provide to implement taps against to get feedback on whether the interface makes sense.
pg_integration/pgConfig.json
describes all the types needed to implement a postgres tap. types prefaced with "standard" are types that will be used in ALL taps. types prefaced with "postgres".pg_integration/src/gen/postgresTap.ts
are the typescript types generated frompg_integration/pgConfig.json
. used https://app.quicktype.io/ to generate the code from the json-schema formatted json.pg_integration/src/app.ts
describes the generic interface that will be used for each tap. then it also shows what that interface looks like when implemented for postgres. also added an example of what it might look like to have a generic singer tap so that the singer internals don't need to be implemented multiple times. in the case where we are trying to use an existing singer tap we just need to implement converters to and from singer and our types.