Support a way to install collections from a static http:// or file:// uri (ala, yum, apt, or helm repos)

[alikins]

Feature Request

Support a remote collections repository similar to the package manager repositories used by yum, apt, helm, etc.

This would provide a way to share collections via a static directory that could be reached via http or local file system. This would be in addition to the REST API based galaxy api and archive download currently used by ansible-galaxy and mazer.

I'll use the term 'yum repo metadata' here, but there are many tools which use a similar approach.

Yum repo metadata style artifact repos has proven to be a extensible and reliable way to distribute content artifacts. Emulating that approach would provide a path for users to provide and manage their own custom content with minimal infrastructure required (a static web server, and a tool to generate the indexes).

Use Case

What problem does this feature solve? Please describe.

Lots of folks want to be able to host ansible content artifacts inside an internal or private network. Some folks want to host their own public content without requiring galaxy.ansible.com to be involved.

Ideally that content would also be easy to mirror and synchronize with common tools and infrastructure.

Content artifacts hosted this way can use common http tooling for caching, both locally and via caching proxies between the client and the primary server. This approach makes it easier to use CDN hosted content artifacts.

Some private content may need additional authentication or authorization to access. Static web content can be easily hosted by http servers that can provide that authentication support.

Users often need to customize and modified and override the source of content. For example, providing newer dev or test versions of content before it is 'production ready'. Or providing customized versions of upstream content and packages. Or maintaining a repo that contains backports of content for older systems.

Ideally the collections repositories could be used from http/https or from a locally mounted file system.

Multiple Collection repositories can be used, typically unioned together.

Proposed Solution

Describe the solution you'd like

Basically, copy yum repo metadata (okay, maybe not the xml part). Or similar tools like https://docs.helm.sh/developing_charts/#the-chart-repository-guide

Create a tool to create/update/manage that data.

At high level:

• some directory
• Has index files with known and predictable names.
• Serialized content artifact data and metadata, likely json or yaml for our case.
• Typically includes a root index that includes names and paths of the entry points to more
  specific data. This will include expected checksums of the other files.
  • Ideally, this root file will be signed with a detached key, so there is a chain-of-trust between the
    root file and anything reference from it.
  • timestamps, versions, checksums, and signatures applied where appropriate to validate content
• Enough data is provided in serialized machine parseable form for client software to discover available content, check for available updates, find and download content artifacts, calculate and resolve any requirements and dependencies, verify and validate content.

Some misc notes:

• some concept of a static repo with an index
  • ideally same format as a content root (~/.ansible/content)
    • needs an index.yml or similar (ie, like yum repomd or helm chart indexes)
      • possibly multiple indexes
        • fqn -> latest version
        • fqn - > versions list
        • (fqn, version) -> collection info
        • galaxy tag -> list of fqn? (fqn, version)
        • platform -> ?
        • content_type -> fqn (ie, 'modules', 'action_plugin', 'roles')
    • collection info and/or indexes will include 'added' date
      • local content root index 'added' represents install date
    • collection index/meta could have a document model change / transaction log
      • ie, each time it is changed, a new transaction record is added
      • transaction record includes
        • seq/inc id number
        • datetimes
        • added / removed
        • user / group / app / process / pid that did the change
        • shasum or index revision id of the 'before' index?

Alternatives

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Implementation

Teachability, Documentation, Adoption, Migration Strategy
If you can, explain the user story, and possibly provide a version of the docs.
Maybe a screenshot or mockup of the design?

[thaumos]

Let's hold on this for the first pass for now.

Our main target for right now is install from Galaxy first.