Run tmt tests

[nikic]

I'm wondering whether we should try to run tmt tests on snapshots. The way I see this working is that after forking into llvm-snapshots we would run the tests for one fedora version in a corresponding container on Github Actions.

Possibly the overall approach could be that we move out the snapshot promotion out of fedora-copr-build into check-todays-snapshot, doing something like this:

• Check whether any build has failed (until now). If so, open an issue. (Same as now.)
• Check whether all builds succeeded. If so, perform the llvm-snapshots fork. (Similar to now, but happens as soon as builds are done, not the next day.)
• Run tmt tests for one configuration. If they fail, open an issue. (Or initially don't do anything, until we know that the tests actually work.)

We could also swap the last two points, which means that we would only fork to llvm-snapshots if tmt tests pass as well. We'd probably need some experience with how often / why they fail first.

[kwk]

@nikic thank you for these ideas.

About tmt tests

Why can't we run tmt tests in the incubator project? Because it is not the final place to consume from? I think for the most part this is negligible.

About forking earlier

I agree that we could potentially fork into the llvm-snapshots on copr repository earlier than at the end of the day. While this sounds nice on paper we have to consider several things:

1. People who consume our snapshots in their CI jobs have worked with this schedule for a longer period of time and are fine with it.
2. If we fork earlier and at a potentially more random point time, we might interrupt ongoing installs.
3. Note that after the fork process we have some time in which the repository metadata on Copr needs to be regenerated. That is an asynchronous and I think long running operation.

I'll try to get more information about the time this all takes in Copr from the #copr team. Then we can decide if the downtime would be acceptable to have during the day.

[nikic]

Why can't we run tmt tests in the incubator project? Because it is not the final place to consume from? I think for the most part this is negligible.

I don't think it matters on which repo they run. I was mainly thinking in terms of how to implement the process so they run exactly once, so doing it during the promotion seemed like a good idea. But if running them on the incubator project is easier that should be totally fine.

As to forking, that's a good concern. I think something to keep in mind is that it's always day somewhere -- the middle of the night for us is probably day time on the west coast. You're right though that at least the current time is very predictable.

[kwk]

I don't think it matters on which repo they run. I was mainly thinking in terms of how to implement the process so they run exactly once, so doing it during the promotion seemed like a good idea. But if running them on the incubator project is easier that should be totally fine.

Okay.

As to forking, that's a good concern. I think something to keep in mind is that it's always day somewhere -- the middle of the night for us is probably day time on the west coast. You're right though that at least the current time is very predictable.

Right, that's the only part I meant. Predictability.

I found out from the #copr team that it can take up to 15 minutes to regenerate the repositories. This is a ticket that may help in the future to inspect status of such actions in Copr: fedora-copr/copr#1108

[kwk]

Sorry for the late update. I didn't follow along our chat internally but apparently the forking takes 1 hour.

[kwk]

@nikic I began working on this here: https://github.com/kwk/llvm-daily-fedora-rpms/blob/main/.github/workflows/tmt.yml . Once the workflow is free from systematic errors I think we can discuss how and when we wanna trigger it.

[kwk]

Exceeding free execution minutes with Github actions

I gave this some thought yesterday. Looking at the usage time, we currently spend 1:45 hours per tmt run given that we have three strategies in parallel and three Fedora versions we test on:

The realistic case I can think of right now is that we have tmt running for 1:45 each day (1:45h x 31). That makes up a total of 3255 minutes per month.

Github gives us 2000 minutes per month in the free plan:

Given that our current selection process for a stable git commit in the LLVM repo has improved, it is theoretically possible that we have successful builds every day, hence we would run the tmt tests each day.

Proposal

I'd say we carefully select which Fedora version we run the tmt tests on. The architecture is more or less fixed and determined by the github action runner being x86_64. This happens to be the fastest architecture we build for on Copr (due to high performance builders). That means Copr builds on that architecture will always be done first for any given day and we can start tmt tests as early as this architecture is done. We should probably wait for all three supported Fedora versions (currently: fedora-38-x86_64, fedora-39-x86_64, and fedora-rawhide-x86_64) to finish in Copr before we kick off the tmt builds for that platform. Each strategy (currenty: standalone, big-merge, bootstrap) should be ran independently.

Summary

• Continuously check if all builds of a strategy (standalone, big-merge, bootstrap) in an fedora-X-x86_64 are green
• Run tmt workflow for the fedora-X-x86_64 and the given strategy
  • For a failing tmt run we may want to report it in the broken snapshot issue
  • For a successful tmt run we want to label the workflow run and make it identifiable so that we don't start the same tmt run if it already ran.

[nikic]

I believe that limit only applies to private repos.

[kwk]

We do run testing-farm now and I think this issue can be closed. Currently the testing-farm results are not gating things because they need fixing. We're working on having the tests green by default and only report on real errors.

[nikic]

Agree that this can be closed now. Thanks for your awesome work on this, the end result is really great!