Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplification of version bumping #7668

Closed
antixar opened this issue Nov 5, 2021 · 3 comments
Closed

Simplification of version bumping #7668

antixar opened this issue Nov 5, 2021 · 3 comments
Labels
area/connectors Connector related issues autoteam team/extensibility type/enhancement New feature or request

Comments

@antixar
Copy link
Contributor

antixar commented Nov 5, 2021

Tell us about the problem you're trying to solve

Now we force to write every new version to 4 different places:
airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/<connector_uuid>.json
airbyte-config/init/src/main/resources/seed/source_definitions.yaml
airbyte-integrations/connectors/<connector_name>/Dockerfile
airbyte-integrations/connectors/<connector_name>/setup.py
+
And update doc files:
docs/SUMMARY.md
docs/integrations/README.md
airbyte-integrations/builds.md

On my opinion it is tough for support and is the reason of merge conflicts.

Describe the solution you’d like

We can automate updating of these files during the publish command.

  1. The source of truth should be one of the following variants:
  • setup.py file. Certainly these options are available for Python packages but I'm sure that Java has a similar one.
    For this we can use the following package options:
  - name  - <connector_unique_name>
  - version -  <connector_version>
  --long-description -  <connector_doc_filepath> or <docfile_full>
  --classifiers - <connector_grade>
  • Dockerfile. And It will be unified approach for all languages. For this we can use the following labels:
LABEL io.airbyte.name=airbyte/<connector_unique_name>
LABEL io.airbyte.version=<connector_version>
LABEL io.airbyte.grade=<connector_grade>
LABEL io.airbyte.docfile=<connector_doc_filepath>
  1. <connector_uuid> can be generated automatically for all new connectors(checking of the source_definitions.yaml file)
  2. <connector_doc_file_path> - should be saved into connectors' folders(e.g.: airbyte-integrations/connectors/<connector_name>/<connector_name>.md) And the publish command will create a symlink to the needed place.
  3. <connector_grade> will be used for updating of docs/integrations/README.md
  4. As result this CI job will add a new commit to a relevant branch if needed.

The main benefit is that all developer will support 2 files only (docfile and setup.py/Dockerfile)

And I'm ready to implement this.

@antixar antixar added the type/enhancement New feature or request label Nov 5, 2021
@antixar
Copy link
Contributor Author

antixar commented Nov 10, 2021

Same issue #1866

@sherifnada sherifnada added the area/connectors Connector related issues label Nov 15, 2021
@sherifnada
Copy link
Contributor

sherifnada commented Nov 29, 2021

I suggest we use the dockerfile as the single source of truth. and btw we don't need to do this for the .json file as they are no longer needed to be updated (they are just kept for backwards compat on older airbyte versions, will be removed eventually)

@girarda
Copy link
Contributor

girarda commented Dec 22, 2022

the flow improved a lot over the last year

@girarda girarda closed this as completed Dec 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues autoteam team/extensibility type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants