-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
migrate greenhouse to config-based cdk #15344
Conversation
/test connector=connectors/source-greenhouse
Build FailedTest summary info:
|
/test connector=connectors/source-greenhouse
Build FailedTest summary info:
|
/test connector=connectors/source-greenhouse |
/test connector=connectors/source-greenhouse |
1 similar comment
/test connector=connectors/source-greenhouse |
/test connector=connectors/source-greenhouse
Build FailedTest summary info:
|
/test connector=connectors/source-greenhouse
Build FailedTest summary info:
|
/test connector=connectors/source-greenhouse |
connection: | ||
- config_path: "secrets/config.json" | ||
status: "succeed" | ||
- config_path: "secrets/config_users_only.json" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these credentials don't work, but I confirmed there are users
records in the account configured in "secrets/config.json"
assert records == [] | ||
|
||
|
||
def test_parse_response_invalid_content(applications_stream): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is not currently handled by low-code, but I'm not sure that
- this ever happens
- Ignoring bad responses is always the right thing to do (probably not)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this test was created as part of the very first PR introducing this connector. So I"m not sure why it's here. Fine by me to remove.
@@ -4,10 +4,12 @@ connector_image: airbyte/source-greenhouse:dev | |||
tests: | |||
spec: | |||
- spec_path: "source_greenhouse/spec.json" | |||
backward_compatibility_tests_config: | |||
disable_for_version: "0.2.7" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need to disable this test because the spec is updated
"properties": { | ||
"api_key": { | ||
"title": "API Key", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated the spec because the linter was complaining
/test connector=connectors/source-greenhouse
Build PassedTest summary info:
|
assert records == [] | ||
|
||
|
||
def test_parse_response_invalid_content(applications_stream): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this test was created as part of the very first PR introducing this connector. So I"m not sure why it's here. Fine by me to remove.
inject_into: "path" | ||
url_base: "*ref(definitions.requester.url_base)" | ||
applications_stream: | ||
type: DeclarativeStream |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we need type: DeclarativeStream
? can't we use a default impl since this is probably the only type of stream we'll have for a while?
name: "demographics_answer_options" | ||
primary_key: "id" | ||
schema_loader: | ||
$ref: "*ref(definitions.schema_loader)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's interesting how verbose / repititive this defintiion is for every stream. It would be really nice if we can DRY this by having a base stream extended by all of these e.g:
base_stream:
type: DeclarativeStream
$options:
primary_key: "id"
schema_loader:
$ref: "*ref(definitions.schema_loader)"
retriever:
$ref: ""
.....
and then each stream just configures the specific option they need
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes!
I DRYed them config
|
||
check: | ||
type: CheckStream | ||
stream_names: ["applications"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Greenhouse is a permissioned system, and the user may only have access to certain streams based on the scopes available to them: https://support.greenhouse.io/hc/en-us/articles/115000521723-Manage-Harvest-API-key-permissions
This check is no worse than the current one. But it would be interesting to think through what it would take to support this in the CDK. maybe it requires something like #14908
/test connector=connectors/source-greenhouse
Build FailedTest summary info:
|
This reverts commit b4de8d6.
/test connector=connectors/source-greenhouse
Build PassedTest summary info:
|
/test connector=connectors/source-greenhouse
Build PassedTest summary info:
|
/publish connector=connectors/source-greenhouse
if you have connectors that successfully published but failed definition generation, follow step 4 here |
What
Migrate the greenhouse connector to config-based
How
YamlDeclarativeSource
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/greenhouse.yaml
Recommended reading order
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/greenhouse.yaml
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/source.py
airbyte-integrations/connectors/source-greenhouse/unit_tests/test_streams.py
airbyte-integrations/connectors/source-greenhouse/source_greenhouse/spec.json
airbyte-integrations/connectors/source-greenhouse/acceptance-test-config.yml
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changesTests
Unit
Put your unit tests output here.
Integration
Put your integration tests output here.
Acceptance
Put your acceptance tests output here.