Add a bundle package and environment to build a basic "externals" stack

[tmadlener]

BEGINRELEASENOTES

• Add a key4hep-externals bundle package that can serve as a minimal external base on top of which most of the Key4hep packages can be built.
  • Most importantly this should include all the dependencies necessary for Gaudi, podio and DD4hep.
  • An additional constraint is given by a roughly 6 hour build time on github hosted runners, which makes it possible to build this into an external base image for further layering.

ENDRELEASENOTES

This is a proposal to add a BundlePackage and an environment that can be used to build a minimal external base on top of which most (all?) Key4hep packages can be built. The current dependencies are mainly collected from trying to build Gaudi and podio on top of it, there are almost certainly packages missing for other "low level" Key4hep packages (e.g. I haven't checked DD4hep).

One of the potential use cases for this is container building where a basis container can be built up-front and only has to change infrequently, while nightly builds or release build without changes to this basis container can be layered on top of it. We (well mostly @madbaron) have been exercising this workflow a bit for building containers for the muon collider, and so far it works out quite well.

I am opening this PR for a general discussion to see what others think and also to see which packages should still go into this package.

The naming is also still up for debate, currently its base but it could just as well be external or minimal if we want to reserve base for something that already has the main building blocks of Key4hep as well (e.g. k4FWCore, DD4hep, EDM4hep, maybe k4geo).

[madbaron]

Just one note to keep in mind if we end up changing the packages included in "base": to keep the multi-step image build working on the github runners, we should keep the compilation time (including setup, checkout and upload to the registry) of each image/layer below 6 hours.

The current set takes about 5h 40m (see for example https://github.com/madbaron/mucoll-spack/actions/runs/12018805366)

[tmadlener]

@madbaron quick heads up that I have renamed the package (and the environment) to use external instead of base. So MuonColliderSoft/mucoll-spack#17 will require some changes.

As dicussed on some occasions, we would like this to be added here, and we would also take on the maintenance for this. We also plan to port some of the muon collider specific github actions that build an image from this to here in a follow up PR unless something speaks strictly against that.

I am not sure if / which CI checks we should / could already add now.

[tmadlener]

A minimal CI check could be to concretize the key4hep-dev-external environment. However, that doesn't tell us too much yet other than very obvious conflicts introduced by the dedicated package configuration. I would be in favor of adding that later once we also have some of the rest of the envisaged workflow ready so that we can do more meaningful checks.

@jmcarcell anything from your side still before we can merge this?

[jmcarcell]

+root7 is not a default one and is probably needed in some places, others may also be needed to prevent a dependency from needing another ROOT.

[tmadlener]

But this should be covered by including the common packages.yaml, I think.

[jmcarcell]

I wonder if changing the requirements in the common packages.yaml and affecting the external stack will be a problem or not 🤔

[tmadlener]

Good question. I think it shouldn't since the ones specified in the external environment take precedence. However, if the requirements in the common packages are underspecified, then building a key4hep-stack on top of the external environment might lead to some packages being rebuilt for the latter because not all variants are present.

This might be something that could be checked in CI (maybe in a follow up PR): concretize this external environment and either the release or nightly environment and see if the packages that are common to both get concretized the same.