Investigate better ways for storing the resources of regression tests

[vkarak]

The current way of storing resources in an external directory is not good, because the files are untracked. Sharing them is not easy either. We should find a different and more efficient way to do it.

[vkarak]

@teojgo Any ideas on this one?

[teojgo]

Yes I am looking in artifact management in general

[vkarak]

Whenever you have something more concrete, feel free to share here.

[vkarak]

This should also cover storing of test artifacts, which now go to the output directory.

[teojgo]

So my idea from a user point of view would be to be able to specify artifacts at the test level as follows:

self.artifacts = [
  ('artifact1', 'upload', ['my_jfrog', 'my_nexus']),
  ('artifact2', 'download', ['my_jfrog'])
]

In the above case, artifact1 would be uploaded to my_jfrog and my_nexus while artifact2 would be donwloaded from my_jfrog. Each of the my_jfrog and my_nexus would have to be set inside the ReFrame configuration, where they type of the repository/instance should be specified (e.g JFrog Artifactory, Sonatype Nexus etc). From there, ReFrame would know which underlying artifact repository type to use which should be implemented (it would look similar to the various module systems) so the corresponding commands to download/upload the artifacts would be generated.

[vkarak]

What about having functions instead?

self.push_artifact(self.stdout, ['repo1', 'repo2'])
self.pull_artifact('repo', 'foo.txt')

• The pull_artifact() will always pull the artifacts in the stage directory.
• These functions would allow repo-specific options to be passed as kwargs.

[vkarak]

Also what about the configuration? How should it look like?

'artifact_repositories': [
    {
        'name': 'my-project-repo',
        'type': 'git',
        'repo': 'https://gitlab.com/group/projectA',
        'options': ['...']
    },
    {
        'name': 'my-other-repo',
        'type': 'jfrog',
        'repo': 'url',
        'options': ['...']
    }
]

[teojgo]

Yeah, typically gou give the name of the repo to the corresponding cli. So assuming that you have configured first with the corresponding url, you refer to it using a chosen name.

[jjotero]

Perhaps solved with #1577