feat(crate): introduce crate concept.

[Zenithar]

Context

• For me, the secret's management is heavily linked to configuration management. I'm preparing a technical move to migrate to configuration management.
• Try to break the Vault dependency.
• Implement the legacy harp registry concept using OCI registry.

Golden circle

Why

• Harmonize configuration deployment to the same way we deploy code.
• Decouple Vault usages – Harp uses Vault for Bundle storage and secret storage, but Harp bundle lifecycle is not bound to a secret storage, and this usage make Vault as a critical component for harp usages.

What

• Harp Bundles contains secret management related information such as labels, annotations, patches, generation templates, validation policies, etc. This information must be securely stored and retrievable.
• Harp should use a persistent Bundle storage where all resources are embedded in one manageable file structure. This file structure will be called a crate. #YetAnotherContainerSynonym

How

• A crate contains one and only one sealed container which can be unsealed by multiple crate consumers.
• A crate can contain multiple archives (templates, patch, or policy collection)
• By using OCI registries to store and retrieve crates, we offer a generic way to handle content already used to deploy code. The idea is to ignite a configuration container concept.
• Reuse code container registry and infrastructure settings (no additional network configuration required to access secret storage).
• Authentication/Authorization is delegated to the registry implementation.
• Integrity/Confidentiality properties are controlled by the sealing process and optional pre-sealing value encryption, to decouple the ability to unseal and the ability to read the secret value.
• Registry event webhooks can be used to trigger external processes (Bundle refresh, etc.).
• All OCI based tools can be used (replication, signing, notarization, provenance authorization, etc.)

Use cases

• Exchange bundles via registry (store the bundle state)
• Generate final configuration from a crate with a valid identity authorized by encryption to unseal the container and use an attached configuration archive as a template configuration file system. (ie. Act as a kubernetes projected volume provider for configMap)

Visual overview

Samples

Sample Cratefile - Used to define the crate OCI layers.

# Required
container "region-eu" {
    # Path to the container.
    path = "region-eu.bundle"
​
    # Idetities for sealing purpose if the container is not sealed.
    identities = [
        "v1.ipk.7u8B1VFrHyMeWyt8Jzj1Nj2BgVB7z-umD8R-OOnJahE", # Security public key
    ]
}
​
# Optional archive layer
# All files matching the filter will be compressed as a tar.gz and embedded
# in the crate.
archive "production" {
    # Root path from where files are crawled to create the archive.
    root = "./production"
​
    # Include filters.
    includes = [
        "**"
    ]
​
    # Exclude filters.
    excludes = [
        "**.go"
    ]
}

Build and push the crate

$ harp crate push --in Cratefile --to registry:ghcr.io/elastic/harp --ref region-boostrap:v1

[Zenithar]

[Zenithar]

OCI image now supports Archives.