Add initial automate-collections-updates.py #113
Conversation
spetrosi
force-pushed
the
automate-collections-updates
branch
from
749e99c to
425f91b
Compare
|
I am pushing here the script that I wrote for automating collections updates based on what Rich wrote in |
|
The script works for me locally, and the next step would be to deploy it somewhere. The system must have access to rhpkg or brew commands, I do not know how to implement this now. I am testing the script by running it with my rhpkg. Any ideas? |
|
@spetrosi, looks very useful! I do not have an answer to your auth question, but I'd believe this pr/113 (well, the subset of this pr :) should be able to replace this code https://src.fedoraproject.org/rpms/linux-system-roles/pull-request/38# in the spec file(s). Don't you think it's cleaner to reduce automate-collections-updates.py to fit in the spec file, eliminate the pr/38 change from the spec file and use automate-collections-updates.py in the build? |
| """Upload new sources""" | ||
| os.chdir(repo) | ||
| subprocess.run( | ||
| [pkg_command, "upload", " ".join(collection_tarballs.values())] |
There was a problem hiding this comment.
We only upload to centos stream or rhel - we don't use vendoring on Fedora.
| cmd = [ | ||
| "ansible-galaxy", | ||
| "collection", | ||
| "install", |
There was a problem hiding this comment.
Can you use download instead of install, to directly download a .tar.gz file? Or is it that the logic that figures out what is the latest release only works with install and not download?
There was a problem hiding this comment.
ansible-galaxy collection cannot download tar.gz only. The best it can do is the following
-n, --no-deps Don't download collections listed as dependencies.
So the script needs to install the collection and then tar it. Luckily, the format of the collection installed through ansible-galaxy is the same as in the tar downloaded from AH via UI.
There was a problem hiding this comment.
? I'm using this logic in another script and it works fine:
ansible-galaxy collection download -vv -p . ansible.posix
e.g.
Starting collection download process to '/home/rmeggins/rh-dist-git/tests/rhel-system-roles/Library/basic/ansible.posix'
Downloading https://galaxy.ansible.com/download/ansible-posix-1.3.0.tar.gz to /home/rmeggins/.ansible/tmp/ansible-local-36821042yavan3/tmpz01gfpll/ansible-posix-1.3.0-tmwjrmic
Downloading collection 'ansible.posix:1.3.0' to '/home/rmeggins/rh-dist-git/tests/rhel-system-roles/Library/basic/ansible.posix'
Collection 'ansible.posix:1.3.0' was downloaded successfully
Writing requirements.yml file of downloaded collections to '/home/rmeggins/rh-dist-git/tests/rhel-system-roles/Library/basic/ansible.posix/requirements.yml'
this creates the file ansible-posix-1.3.0.tar.gz
There was a problem hiding this comment.
Yuy, it does work with ansible-galaxy from ansible-core, I'll use it here.
| # TODO: Share build_url somewhere | ||
| finally: | ||
| print("Cleaning up the downloaded repository") | ||
| os.chdir("..") |
There was a problem hiding this comment.
which directory is this script supposed to be run from? I think you'll have to be careful about paths if you assume a certain directory, or don't assume a cwd anywhere - all of the open calls should construct the full path and filename - use the cwd argument of subprocess.run to run the command in a certain directory.
There was a problem hiding this comment.
Good point, I'll fix this
| for value in coll_content_values: | ||
| changelog = yaml.safe_load( | ||
| open(os.path.join(coll_dir, "changelogs/changelog.yaml")) | ||
| ) |
There was a problem hiding this comment.
do we need to look for updates in the changelog? Can't we just assume if the version is different, it is newer, and use it? I guess that would cause problems if they updated the version a lot for trivial, non-API issues, but I don't think that is the case.
There was a problem hiding this comment.
I pushed this as a suggestion for you to look at. And now with your comment I understand that it is a loss of time to check for the changelog. It is not a big deal for us to vendor a new version if it got released. I'll remove this logic.
No, I don't think the specfile should be pulling collection tarballs from galaxy and automation hub. I think that should be a separate process. Pulling sources in the specfile is against the RPM way, which is to list them in the |
For centos, there is the |
spetrosi
force-pushed
the
automate-collections-updates
branch
from
425f91b to
255219f
Compare
Move var definitions, remove redundant list() and print() Break the code into functions Use centpkg clone rhel-system-roles instead of URL cloning Remove logic that checks changelogs
|
lgtm - just the formatting issue to fix |
No description provided.