Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the release artifacts more granular. #252

Closed
vaikas opened this issue Jul 31, 2022 · 4 comments · Fixed by #268
Closed

Make the release artifacts more granular. #252

vaikas opened this issue Jul 31, 2022 · 4 comments · Fixed by #268
Assignees
Labels
enhancement New feature or request

Comments

@vaikas
Copy link
Contributor

vaikas commented Jul 31, 2022

Description

While it's convenient to do a one fell swoop and launch all the jobs at the same time, it does make things slower because of the exponential back-off that jobs use. Since there are several jobs that get launched simultaneously and because previous jobs need to finish before the final ones do, by the time we get to later stage jobs, they are backing off into the minutes.

So, if we create the release artifacts along the lines of steps as in the README.md and then launch those and wait for them to finish before starting the next step, we could shave off (guesstimate, minutes?) off the startup time.

I think the natural break from README (trillian, rekor, ctlog, fulcio) might make sense and make things more granular. But I think even just pulling the trillian into it's own would be good.
There's this PR that might help with that:
google/trillian#2754

So, if we get that merged in, then we can remove some bits from here. But, this is just keeping track of this thought, as well as depending on how long it takes me to get that in, might start some work here ahead 🤷

@vaikas vaikas added the enhancement New feature or request label Jul 31, 2022
@vaikas
Copy link
Contributor Author

vaikas commented Aug 5, 2022

I started to do this here:
https://github.com/sigstore/scaffolding/blob/main/Makefile#L37

And ideally we'd break this to create multiple artifacts similarly, maybe a for loop here:
for i in [ctlog, fulcio, rekor, tuf, trillian]
ko resolve .... -f config/$i > $i.yaml

And lastly also need to release ./hack/setup-scaffolding.sh like we do ./hack/setup-kind.sh
probably just need to add it here:
https://github.com/sigstore/scaffolding/blob/main/.goreleaser.yaml#L26

These are notes mainly for myself, but if @cpanato you find some time (not a big deal if you can't) to make sure this looks right I'd appreciate it.

@vaikas vaikas self-assigned this Aug 7, 2022
@vaikas
Copy link
Contributor Author

vaikas commented Aug 7, 2022

Little more background on the necessity for doing this. Since switching to using TUF root vs. env variables, we need to do a manual step for copying the various secrets that TUF root needs (ctlog, fulcio, and rekor) to TUF namespace so that it can construct a root from them. We could probably do by creating a bundle with ctlog, trillian, fulcio, and rekor into a single bundle, but coupled with the speed ups that come from doing things in a sequence this makes sense to do it one by one.
To keep the convenience of running a simple command, there's a hack/setup-scaffolding.sh script that installs the scaffolding in the correct sequence to make it as fast as possible, copying the TUF secrets and then starting TUF.

@cpanato
Copy link
Member

cpanato commented Aug 8, 2022

you find the time before me :)

@vaikas
Copy link
Contributor Author

vaikas commented Aug 8, 2022

LOL, for once :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants