Add dev.eessi.io support including project subdirectories

[Neves-P]

This PR replaces open WIP PR #804 (to merge from a more up to date and tested branch) and adds support for building into the dev.eessi.io repository.

Documentation for how to install into dev.eessi.io is available in https://www.eessi.io/docs/adding_software/adding_development_software/, with maintainer documentation to come in EESSI/dev.eessi.io-example#12

With this PR, software builds are installed into /cvmfs/dev.eessi.io/2023.06/$EESSI_DEV_PROJECT/. $EESSI_DEV_PROJECT is determined automatically during the build process and comes from the name of the GitHub repository that the bot is listening according to the format: dev.eessi.io-EESSI_DEV_PROJECT. See: https://github.com/EESSI/dev.eessi.io-example

Existing builds to dev.eessi.io were tested with the code that is about to be merged as verified in EESSI/dev.eessi.io-example#14

[Neves-P]

Talked with @bedroge about this and decided to try just adding a simple if statement to EESSI extend that would account for builds targeting the development repository, changes in: ee2bcb1

I'd like to clarify to things:

1. How to test this without potentially breaking working code. I suppose that the change to the EESSI-extend module needs to actually be ingested first so that the new module file is used?
2. Does this actually work, or is there a better or improved way of achieving this.

@ocaisa , would you have any suggestions regarding this?

PS: I'm marking this a draft to prevent accidental merges as this could break production code. This PR also got extremely long with many commits reverting small changes and applying them again. I'm happy to open a new one from a clean branch to prevent polluting the repo's commit history with many trial and error commits.

[ocaisa]

Talked with @bedroge about this and decided to try just adding a simple if statement to EESSI extend that would account for builds targeting the development repository, changes in: ee2bcb1

In general, I think this is the way to go. I would like to target the dev repos just the same as we do project extensions. There will need to be a step that generates the path with the subdirectory, but that could all be done with a single environment variable (if the variable EESSI_DEV_PROJECT is set, we are targeting /cvmfs/dev.eessi.io and should use the value of the variable to decide the subdirectory). That would mean you set the variable to the project and there is only a very small amount of special treatment to determine EESSI_PROJECT_INSTALL

I'd like to clarify to things:

1. How to test this without potentially breaking working code. I suppose that the change to the EESSI-extend module needs to actually be ingested first so that the new module file is used?

EESSI-extend is just an easyconfig that creates a module file, you can install it locally.

2. Does this actually work, or is there a better or improved way of achieving this.

@ocaisa , would you have any suggestions regarding this?

Rather than complicate our lives with

/cvmfs/dev.eessi.io/versions/2023.06/example/software/linux/aarch64/neoverse_n1

perhaps we should be using

/cvmfs/dev.eessi.io/example/versions/2023.06/software/linux/aarch64/neoverse_n1

then you have direct mapping /cvmfs/software.eessi.io -> /cvmfs/dev.eessi.io/example

[Neves-P]

Thanks @ocaisa , indeed that makes it a lot simpler.

In fact, I think we don't need to even touch EESSI-extend (or just tweak the check to see if the project directory exists), because we should be able to just pass /cvmfs/dev.eessi.io/example as EESSI_PROJECT_INSTALL in one go from theEESSI-install.sh. I'll give this a try now on the example repo.

[ocaisa]

Ditto here, you could just tweak the definition of EESSI_CVMFS_REPO_OVERRIDE

[Neves-P]

Agreed, but I think we need to set $EESSI_CVMFS_REPO_OVERRIDE before we call minimal_eessi_env, in bot/build.sh(?). I'm trying this now in acb896a

[Neves-P]

Not there yet, the check for software_dir_overlay fails in create_tarball.sh but should be close to sorted. I'll pick this up tomorrow.