[skip-ci] Packit/TMT: Run gating tests

[lsm5]

Commit 1
PR Labels: apply release label to release- branch PRs

This will be useful in the followup commit that enables TMT test jobs on PRs.

PRs on main branch should be tested with bleeding-edge dependencies from the podman-next COPR while PRs on release branches should be tested only with the official distro packages. Packit will run/skip the relevant set of tests based on this label.

Commit 2
Packit/TMT: Run system tests

This commit enables TMT jobs triggered by Packit to run system tests.

2 set of jobs dev and release have been added. dev jobs are meant to run on main PRs with additional package updates fetched from podman-next copr while release jobs are meant to run on release- branch PRs using only the dependencies present in the official distro.

Packit checks PR labels (see previous commit) to filter out the jobs that get run.

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[mtrmac]

@lsm5 are you still working on this?

(If so, this is not urgent from my POV, no need to push it to the front of your queue — I’d just like to close the PR if there is absolutely no chance this work will continue.)

[lsm5]

@mtrmac I do plan to work on this. But may take a while. I have bookmarked it for myself so I don't lose track. So please feel free to close it.

[mtrmac]

Thanks, if it is still relevant, let’s keep it open. It might help, or inspire, other contributors.

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[github-actions]

A friendly reminder that this PR had no activity for 30 days.

[lsm5]

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ? Asking mainly RE: packit + tmt tests.

[mtrmac]

@edsantiago is authoritative for system tests (question above).

[edsantiago]

Forgive me please, I don't understand the question. It's been a while since I've looked at skopeo gating tests, but my recollection is that they run on a plain system, via dnf install skopeo-tests. What's the context for "in a container"?

[mtrmac]

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ?

IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.

I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.

[thrix]

/packit test

[thrix]

So, unfortunately, the issue we are hitting is a bug in Packit:

packit/packit-service#2028

It was not considered a bug before, as people were not hitting it and we did not know it had this consequence.
They are working on a fix.

[lsm5]

So, unfortunately, the issue we are hitting is a bug in Packit:

packit/packit-service#2028

It was not considered a bug before, as people were not hitting it and we did not know it had this consequence. They are working on a fix.

Thanks @thrix . I'll disable the rhel tests for now.

[lsm5]

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ?

IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.

I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.

ack, the test-system target seems to spawn some container too. Anyway, I'll try to mimic the cirrus setup in tmt. Thanks

[packit-as-a-service]

Failed to load packit config file:

Please correct data and retry.

For more info, please check out the documentation or contact the Packit team. You can also use our CLI command validate-config or our pre-commit hooks for validation of the configuration.

[packit-as-a-service]

Failed to load packit config file:

Please correct data and retry.

For more info, please check out the documentation or contact the Packit team. You can also use our CLI command validate-config or our pre-commit hooks for validation of the configuration.

[packit-as-a-service]

Tests failed. @containers/packit-build please check.

[lsm5]

@mtrmac RE: your earlier questions:

What is the longer-term plan? Is this going intended to be in addition to .cirrus.yml, or eventually replace it?

Plan is to eventually get rid of cirrus and depend only on TMT assuming we'll have all functionality sooner or later via TMT.

Update: I have pretty much redone the whole PR with scope limited to only triggering system tests as we'll need those for official Fedora and CentOS Stream gating too. So, I feel best to get those in first. Please also check out the updated PR description ^ for details on the 2 included commits.

You'll see a bunch of new jobs in CI. The ones with suffix dev will run on main branch. The ones with suffix release will show as Queued but are expected to not run on main. Reverse should happen on release-* branches once this change lands there.(Untested on release branches but I guess we'll find out when we cherrypick).

• systemtest/tmt/* lists the actual test and the script.
• plans/main.fmf configures the test environment based on the initiator (packit, fedora ci, human etc.). Any filtering of tests to run based on the environment will also be done here. For example, see the same on container-selinux

In this PR, I'm no longer changing the Makefile or scripts in hack/. So, that should make things a lot simpler.

I'll work on the integration / lint and ostree tests in followup PRs. I'll keep your other comments here in mind while I work on those later.

[lsm5]

Another advantage of TMT is convenient reverse dependency testing. See this c/common PR. That will run the tests maintained in skopeo on every PR in c/common.

[lsm5]

/packit test