Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script for bumping connectors version #1866

Closed
1 of 3 tasks
eugene-kulak opened this issue Jan 27, 2021 · 3 comments
Closed
1 of 3 tasks

Add script for bumping connectors version #1866

eugene-kulak opened this issue Jan 27, 2021 · 3 comments
Assignees
Labels
area/connectors Connector related issues autoteam build DX priority/low Low priority team/extensibility technical-debt issues to fix code smell type/enhancement New feature or request

Comments

@eugene-kulak
Copy link
Contributor

eugene-kulak commented Jan 27, 2021

Tell us about the problem you're trying to solve

Currently, every PR that modifies connectors code need to do a separate bump commit after all CR passed and all tests are green because CI action checks if docker images with a specific tag/version exist.

Describe the solution you’d like

To have a command similar to test connector=file-source, something like
bump connector=file-source

Describe the alternative you’ve considered or used

Manual edit of 3 files:
airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/<uuid>.json
airbyte-config/init/src/main/resources/seed/source_definitions.yml
airbyte-integrations/connectors/<connector>/Dockerfile

Checklist

  • Add script to bump connector's version - 3h
  • Add GitHub action that runs this script - 1.5h
  • Add script to commit changed files under CI account - 2.5h

┆Issue is synchronized with this Asana task by Unito

@eugene-kulak eugene-kulak added type/enhancement New feature or request zazmic labels Jan 27, 2021
@eugene-kulak eugene-kulak self-assigned this Jan 27, 2021
@sherifnada sherifnada added this to the Zazmic: 02/19/2021 milestone Feb 5, 2021
@sherifnada sherifnada changed the title Introduce github bot command for bumping connectors version Add script for bumping connectors version Jun 2, 2021
@sherifnada sherifnada added the area/connectors Connector related issues label Jun 2, 2021
@sherifnada sherifnada removed this from the Connectors June 11, 2021 milestone Jun 10, 2021
@sherifnada sherifnada removed the zazmic label Aug 6, 2021
@avida avida self-assigned this Sep 21, 2021
@avida avida added this to the Connectors, October 1st 2021 milestone Sep 30, 2021
@sherifnada sherifnada added the priority/low Low priority label Oct 29, 2021
@antixar
Copy link
Contributor

antixar commented Nov 10, 2021

Same issue #7668

@keu
Copy link
Contributor

keu commented Nov 10, 2021

I was under impression that the following script was merged to master...

import argparse
import json
import sys
from pathlib import Pathimport yaml
​
​
HERE = Path(__file__).parent.absolute()
​
SOURCE_DEFINITIONS_PATH = HERE / 'airbyte-config/init/src/main/resources/seed/source_definitions.yaml'
UUID_PATH = HERE / 'airbyte-config/init/src/main/resources/config/STANDARD_SOURCE_DEFINITION/{}.json'
CONNECTOR_PATH = HERE / 'airbyte-integrations/connectors/{}/Dockerfile'
​
​
def bump_version(version: str, kind: str) -> str:
    major, minor, patch = list(map(int, version.split('.')))
    if kind == 'major':
        major += 1
        minor = 0
        patch = 0
    elif kind == 'minor':
        minor += 1
        patch = 0
    elif kind == 'patch':
        patch += 1return '.'.join(list(map(str, (major, minor, patch))))
​
​
def main():
    parser = argparse.ArgumentParser(add_help=False)
    parser.add_argument("source_name", type=str, help="Source name")
    parser.add_argument("bumping_type", type=str, choices=['major', 'minor', 'patch'], help="Part of version to bump")
    args = parser.parse_args()
​
    with open(str(SOURCE_DEFINITIONS_PATH), 'r+') as f:
        source_definitions = yaml.safe_load(f)
        for source in source_definitions:
            if source['dockerRepository'] == 'airbyte/' + args.source_name:
                uuid = source['sourceDefinitionId']
                old_version = source['dockerImageTag']
                new_version = bump_version(old_version, args.bumping_type)
                source['dockerImageTag'] = new_version
                break
        else:
            raise ValueError(f'Connector {args.source_name} not found in {SOURCE_DEFINITIONS_PATH}')
​
        print(f'Found connector version {old_version}, bumping to {new_version}')
        f.seek(0)
        f.write(yaml.dump(source_definitions, sort_keys=False))
​
    with open(str(UUID_PATH).format(uuid), 'r+') as f:
        uuid_file = json.load(f)
        uuid_file['dockerImageTag'] = new_version
        f.seek(0)
        json.dump(uuid_file, f, indent=2)
​
    with open(str(CONNECTOR_PATH).format(args.source_name), 'r+') as f:
        content = f.read()
        content = content.replace(f'io.airbyte.version={old_version}', f'io.airbyte.version={new_version}')
        f.seek(0)
        f.write(content)
​
​
if __name__ == "__main__":
    sys.exit(main())

@sherifnada sherifnada removed this from the Connectors Nov 26 2021 milestone Nov 15, 2021
@girarda girarda added the technical-debt issues to fix code smell label Dec 22, 2022
@maxi297
Copy link
Contributor

maxi297 commented Jan 31, 2023

@maxi297 maxi297 closed this as completed Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues autoteam build DX priority/low Low priority team/extensibility technical-debt issues to fix code smell type/enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

8 participants