Testing Quarkus against different versions of GraalVM/Mandrel

[zakkak]

Description

It looks like our current way of testing Quarkus with different GraalVM CE and Mandrel versions is a bit flawed and might let issues go through.

I have shared this in the quarkus-dev mailing list as well, but having it here makes it easier to link issues and PRs.

Existing pipelines

I am personally aware of the following pipelines (please let me know if there are more out there):

1. Quarkus Upstream [1]

• Tests latest Quarkus with latest supported GraalVM CE (21.0.0 ATM) in a builder container.
  • Runs on PRs (multiple times a day)
• Tests Quarkus releases that are still getting backports with the corresponding supported GraalVM CE (e.g. 20.3 for 1.11) in a builder container.
  • Runs on PRs (rarely)

2. Mandrel Upstream

• Tests Mandrel built from source with the corresponding Quarkus releases that the version at hand supports, e.g.:
  • Mandrel 20.3 is tested with Quarkus 1.11 release and Quarkus main branch [3]
    • Runs on PRs (which are rare, and no re-runs after release)
  • Mandrel 21.0 is tested with Quarkus latest release and Quarkus main branch [4]
    • Runs on PRs (which are rare, and no re-runs after release)
  • Mandrel "master" (mandrel build of latest graal) is tested with Quarkus latest release and Quarkus main branch [5]
    • Runs nighlty

3. Mandrel's public Jenkins instance [6]

• Our most complete set of pipelines
• Tests both Linux and Windows
• Tests both docker and podman
• Tests both builder images and hosted
• Tests same version pairs like Mandrel Upstream
• In addition to Quarkus ITs it also runs https://github.com/Karm/mandrel-integration-tests

• Runs on Mandrel code change (polls github for changes) which is rare

4. Graal Upstream [7]

• Tests latest Graal build from source with Quarkus latest release
  • Runs nightly

5. Foivos' workflow [8]

• Tests any version of GraalVM CE built from source with any version of Quarkus build from source. This is actually an augmented version of the Graal Upstream workflow
  • Runs on demand

People managing the pipelines

• Quarkus Upstream: Quarkus community
• Mandrel Upstream: Mandrel community (@zakkak mostly)
• Mandrel's public Jenkins instance: @Karm and partly @zakkak
• Graal Upstream: @zakkak
• @zakkak's workflow: @zakkak (it's only a couple of weeks old though)

People monitoring the pipelines

• Quarkus Upstream: Monitored by the Quarkus community
  (mostly @gsmet, @gastaldi, and @geoand if I am not mistaken)
• Mandrel Upstream: Monitored by the Mandrel community (@zakkak mostly)
• Mandrel's public Jenkins instance: Monitored by @Karm and partly
  by @zakkak
• Graal Upstream: Monitored by the GraalVM community (@zakkak mostly if not solely :) )
  Note that most often the issues that pop up need to be resolved on the Quarkus side, so the GraalVM community can't do much about it.
• @zakkak's workflow: @zakkak (it's only a couple of weeks old though)

Issues with the current state

1. Testing is happening in multiple places, so it's hard to get the full picture.
2. Our most complete set of pipelines is in Mandrel's public Jenkins instance, which unfortunately runs on limited resources and is thus not a good fit for scheduling all the tests we need. Mandrel's public Jenkins instance is meant to be used for building and testing upstream Mandrel releases and not for continuous testing.
3. We are missing test-cases/scenarios:
   1. Mandrel Quarkus builder image is not tested by Quarkus upstream (it's only tested by the Mandrel team before Mandrel releases or on demand)
   2. Quarkus latest is not being tested with 20.3.
4. Most pipelines are monitored mostly by the Mandrel team. I believe both teams should keep an eye on the pipelines and depending on the nature of the failure take action or ensure that the right people are aware of the failure.
5. Due to having the pipelines in multiple places it is hard to ensure that they are in sync, and they are actually testing what they should.
6. Native tests take pretty long to run (~2h with 8 parallel executors). We might not be able to do much about this but maybe we can prioritize tests (and even combine some?).

Implementation Ideas

1. Gather the results of all native-image related testing to a single place:

• https://status.quarkus.io/ seems like a good place for this
• Enhance our github pipelines to perform tests previously performed by Mandrel's public Jenkins instance.
• Split the pipelines between Quarkus and GraalVM organization to avoid saturating either.
• Find a way to de-duplicate pipelines
  • In GH we can use scheduled builds and https://github.com/marketplace/actions/workflow-dispatch for triggering builds from other workflows or to perform on-demand parameterized builds like I do in [8]

2. Send notifications to at least one person from the Mandrel team and one person from the Quarkus team when a scheduled workflow fails (failures of jobs triggered by PRs should be treated differently, ideally blocking the corresponding PR).
3. Reduce the coverage of (or retire entirely) the Upstream Graal pipeline [7]. My experience so far is that the graalvm community is not really paying attention to it and keeping it there just adds complexity (since we need to keep it in sync and wait for someone from the graalvm team to review and merge changes, which is understandably not a high priority for them)
4. We might want to come with a subset of native tests and configurations that we want to run on PRs or some other events and only run the full set once per week or on some other schedule.
5. Explore the potential of combining the tested functionality of multiple native integration tests in a single native image. E.g., for gRPC we currently build 8 native images. Can we have the same coverage with less native images? Discussed in CI: Reduce time spent in native tests by grouping more tests in the same native image #16439

Please share your opinion, ideas, and any potential info I might have left out.
Thanks for reading that far!

[1]
https://github.com/quarkusio/quarkus/actions/workflows/ci-actions-incremental.yml
[2]
https://github.com/graalvm/mandrel/blob/mandrel/20.1/.github/workflows/mandrel.yml
[3]
https://github.com/graalvm/mandrel/blob/mandrel/20.3/.github/workflows/mandrel.yml
[4]
https://github.com/graalvm/mandrel/blob/mandrel/21.0/.github/workflows/mandrel.yml
[5]
https://github.com/graalvm/mandrel/blob/default/.github/workflows/mandrel.yml
[6] https://ci.modcluster.io/view/Mandrel/
[7] https://github.com/oracle/graal/actions/workflows/quarkus.yml
[8] https://github.com/zakkak/graalvm-quarkus-ci/actions/workflows/quarkus.yml

[quarkus-bot]

/cc @galderz

[zakkak]

It's been a while, so here is an update :)

Issues with the current state

1. Testing is happening in multiple places, so it's hard to get the full picture.

Currently the easiest way to monitor things is by inspecting Mandrel's Nightly CI runs.

2. Our most complete set of pipelines is in Mandrel's public Jenkins instance, which unfortunately runs on limited resources and is thus not a good fit for scheduling all the tests we need. Mandrel's public Jenkins instance is meant to be used for building and testing upstream Mandrel releases and not for continuous testing.

At the moment Mandrel's public Jenkins instance runs about the same tests as Mandrel's Nightly CI runs.

3. We are missing test-cases/scenarios:
   1. Mandrel Quarkus builder image is not tested by Quarkus upstream (it's only tested by the Mandrel team before Mandrel releases or on demand)
   2. Quarkus latest is not being tested with 20.3.

There are now Mandrel github jobs for testing these (and more) nightly.

4. Most pipelines are monitored mostly by the Mandrel team. I believe both teams should keep an eye on the pipelines and depending on the nature of the failure take action or ensure that the right people are aware of the failure.

This is still an issue. The main monitoring is performed by @zakkak and @Karm . This seems to work fine so far, but a couple of extra eyes would be nice to have.

5. Due to having the pipelines in multiple places it is hard to ensure that they are in sync, and they are actually testing what they should.

This seems to be fixed by a combination of graalvm/mandrel#331 on the Mandrel side and #13124 on the Quarkus side.

6. Native tests take pretty long to run (~2h with 8 parallel executors). We might not be able to do much about this but maybe we can prioritize tests (and even combine some?).

On the Quarkus side this is mostly fixed by #13243.
On the Mandrel side graalvm/mandrel#331 seems to have helped significantly, in <5h we run 11 different configurations with ~25 groups of native tests each.

[galderz]

Fantastic work @zakkak!! 👏👏👏

[geoand]

Great stuff!

Can we close this issue or is there still more to be done that is tracked by this?

[zakkak]

Can we close this issue or is there still more to be done that is tracked by this?

I think that the only outstanding issue is that we would still need someone from the quarkus team keeping an eye on some of the configurations, but I still haven't figured out a reasonable way to do so without spamming them with countless emails of false positives (due to our aggressive testing we often have plenty of failing tests in the CI ).

Nevertheless I think we can close this issue and when/if I find a solution about that issue I will reach out to the Quarkus team :)