docs: establish Artifacts WG

[afflom]

Establish the Artifacts WG under CNCF TAG App Delivery.

Closes: #368

[afflom]

@joshgav ptal

[joshgav]

@afflom thank you for preparing this charter and WG proposal! 🎉 The charter you proposed includes the proposed purpose, goals, and activities of the WG 👍, and we included initial problem statements/user scenarios as an appendix too to inform reviewers.

Next we'll:

1. Ask TAG App Delivery leaders and TOC members to review and comment on this PR
2. Merge this PR and officially start the WG when we have consensus
3. Select WG leaders, meeting times and other logistics; though we already have folks in mind to nominate.
4. Continue the meetings and work we've already started.

Let's note here that the content of the charter is based on text and discussions in comments from https://docs.google.com/document/d/1w_lo2RZDKeEzQg4DMV-9Tq4ir_znONj_ypJ27CUfMgY/. Folks may want to review those comments to get full background.

As the WG briefly also discussed, we didn't include the list of interested parties from the draft doc so as not to permanently clutter this repo and doc. However, we should think about how to notify and invite them to our meetings and contribute to our work. Perhaps we can CC them here and invite them all to the https://lists.cncf.io/g/cncf-tag-app-delivery/topics listserv and #wg-artifacts Slack channel.

[joshgav]

@afflom, you'll need to attest to the content you submit by signing the commit. You can do this from the git CLI with a command like: git commit -s --amend --no-edit && git push --force.

Otherwise LGTM!

[TracyRagan]

Those changes look good to me.

[afflom]

@joshgav signed-off

[joshgav]

LGTM!

[scottrigby]

Looks great! And thanks for following up after the kccnc meetings. Tl;dr, 🚢 👍

One note: this md doc doesn't include the "existing work" section from the Charter draft, but I think that's fine we can keep a link to that and can fold this info into one of the work products produced by the group.

[scottrigby]

As the WG briefly also discussed, we didn't include the list of interested parties from the draft doc so as not to permanently clutter this repo and doc. However, we should think about how to notify and invite them to our meetings and contribute to our work. Perhaps we can CC them here and invite them all to the https://lists.cncf.io/g/cncf-tag-app-delivery/topics listserv and #wg-artifacts Slack channel.

@joshgav We could revisit including an interested parties doc for each WG in this repo? (Related issue #256 (comment)). Or perhaps it's worth adding a "Working Groups" section to the CNCF Community Groups site so that folks can sign up directly for activity by specific WGs. Either way, something we can follow-up on after this WG is fully established 😸

[SteveLasker]

Thanks @afflom for getting this started.
A few minor nits to generalize the charter.

[SteveLasker]

nit: official is redundant

[afflom]

Agreed

[SteveLasker]

Iana registration feels like a specific thing that might be part of the discussion, but is it a goal to remove?
Iana registration was used to solve a clearing house of ownership for a specific artifactType, It solves the conflicting file extensions type problem. What keeps someone from defining different versions of "helm", or "spdx"?
See Defining a Unique Artifact Type
To avoid bike shedding specifics here, perhaps we can remove line 28, and explore the dynamic discovery?

[afflom]

@SteveLasker, I agree that we should not call out IANA here, however I do think that defining an extensibility model is important for the working group to have as a goal. Ideally, it would take administrative burden off of registry operators and publishers by dynamically registering schemas as they are used to describe artifacts and their relationships.

[ktarplee]

In an effort to not over promise we might want to make a section called "Stretch Goals" to cover topics including decentralization and federation. We could also include as a stretch goal something about how OCI could be a package indices replacement for PyPI, Apt, etc.

[afflom]

@ktarplee , I think that this might be a good approach. Would these stretch goals be encapsulated within the main goals? If so, would they be most appropriate in the charter or within the working group's prototyping space?

[ktarplee]

I think the "package index" idea is covered under the "Search Query API" goal. It might make sense to add "package index" to the "Activities" section where we talk about npm, PyPI, etc. We only discuss the artifacts in that activity (i.e., how we might store them) and not how we might search for them (i.e., the package index functionality).

[joshgav]

To be frank I expect we'll keep static registration but introduce experiments for dynamic type/schema discovery alongside that.

I think we've removed this topic from the charter in any case, LGTM.

[sudo-bmitch]

I'm expecting this repo to be archived in the not too distant future:
opencontainers/artifacts#61
opencontainers/artifacts#63

We may want to link to image-spec instead.

[SteveLasker]

Having this group demonstrates theres been a lot of value in OCI Artifacts, so it's not clear why we would archive the repo.

[sudo-bmitch]

I think that discussion is best had in the linked issues. The short version is that image-spec has added a lot of details on how to package content other than images, and it has made changes that aren't included in the artifact repo. Rather than trying to keep the two in sync, I think it makes sense to consolidate to one repo.

[afflom]

I've removed the reference altogether in my re-imagining of the goals section.

[rchincha]

That,

1. all "artifacts" use OCI artifacts path and by extension use the dist-spec referrers facility

• OR -

2. all "artifacts" use https://github.com/opencontainers/artifacts (or derivative) and/or implement specs coming out of this WG

remains unresolved IMO.

I suspect this may just turn out to be a cost-benefit business decision in the end. Too costly to do 2) so just do 1) for example. But not sure we want to make this assumption so early in the game.

[joshgav]

Thanks for this discussion, it's helpful to understand the potential issues with existing artifacts and the evolution of image-spec. Thanks for participating in this group @sudo-bmitch.

[sudo-bmitch]

There's going to be more than one data model for artifact content. This implies they will all have a single model.

[sudo-bmitch]

If this is getting into defining a new OCI manifest type, OCI is already planning to look at a replacement for the artifact manifest after the image-spec 1.1 release. It's probably better to do that over in OCI.

[afflom]

I think that's the reason that this working group was originally pitched to the OCI maintainers. It would be unfortunate if the next OCI data model was not compatible with the search API. With consideration for that prior attempt at establishing this working group under OCI, do you have any suggestions for combining these efforts or at least ensuring this working group's output is a consideration in the development of the next OCI data model?

[sudo-bmitch]

Our focus right now is on the 1.1 release. Once that's done, merging efforts would be great.

[sudo-bmitch]

This could be rephrased a bit to focus on the need to package various types of artifacts vs defining a single model (depending on the level where we implement this, we may still have a single model, we may leverage an existing model, or there may be multiple models).

Suggested change

	* Contribute to a common data model for Artifacts in OCI
	* Define data models for storing Artifacts in OCI

[AlexsJones]

I would caution activities that define a deliverable outcome as that's what your success will be hinged upon.
Defining OCI for artefacts is going to be a long process of getting people to agree with each other.
Perhaps you can save yourself with wording here

Suggested change

	* Contribute to a common data model for Artifacts in OCI
	Identify an exemplar data model for storing artefacts in OCI

[afflom]

@sudo-bmitch @AlexsJones, based on your suggestions and the feedback from @justincormack, I've tried to better address this in the revised goals.

[sudo-bmitch]

There's dynamically discovering schemas, and dynamically discovering the artifacts themselves. The former may be both more difficult and less useful to end users.

[afflom]

I have thoughts on this, but since the line is removed, I'll save them for when we discuss implementation.

[joshgav]

FWIW there's a comparison to be made with JSON schema and https://www.schemastore.org/json/. VS Code and other tools rely on schemastore for type checks and coding hints; perhaps runtime tools also verify manifests against published schemas. Of course, schemastore is a static registration system like IANA. My point is there does seem to be value in enabling many tools and services to discover the types and schemas behind manifests.

[sudo-bmitch]

Some suggestions based on Justin's feedback.

[sudo-bmitch]

This could be rephrased a bit to focus on the need to package various types of artifacts vs defining a single model (depending on the level where we implement this, we may still have a single model, we may leverage an existing model, or there may be multiple models).

Suggested change

	* Contribute to a common data model for Artifacts in OCI
	* Define data models for storing Artifacts in OCI

[sudo-bmitch]

Consider removing since this covers how rather than why. We may still do this, but it can also be an arbitrary constraint on our work.

[afflom]

Removed

[sudo-bmitch]

I'm trying to figure out how to rephrase this towards the problem being solved vs the how we are solving it. I'm not excited about my phrasing, but I want to get to a search/query API that may need more parameters than exist in current OCI query APIs.

Suggested change

	* Define an Artifact Query API based on specific or common data models as an OCI distribution-spec extension
	* Define an API to query for Artifacts using parameters needed for the different types of Artifact tooling.

[afflom]

I used this as inspo in my goals refactor. Can you PTAL?

[AlexsJones]

Will you really be developing patterns rather than observing and notating what is in use?

[afflom]

I'm hoping that my refactor on the goals section clarifies this a bit. Can you PTAL and let me know if you'd like this bit about patterns reworked?

[joshgav]

@AlexsJones I believe the group intends to influence common patterns directly. First they'll gather up many existing practices, then they'll share practices and patterns across them, and finally they'll influence and standardize these common practices in OCI specifications.

[AlexsJones]

I would caution activities that define a deliverable outcome as that's what your success will be hinged upon.
Defining OCI for artefacts is going to be a long process of getting people to agree with each other.
Perhaps you can save yourself with wording here

Suggested change

	* Contribute to a common data model for Artifacts in OCI
	Identify an exemplar data model for storing artefacts in OCI

[AlexsJones]

This isn't an activity is it?

[afflom]

Good catch. I've removed it.

[afflom]

I've reworked the goals based on all of the feedback here and in the Artifacts WG meetings. In this rework, I've distilled the goals into 3 "stories". Those are research, design, and prototype. Please let me know if these work or if I should revert to the previous goals format.

[sabre1041]

Love the progress the WG has made here. Made a bunch of suggestions based on review

[sabre1041]

Suggested change

	* As an artifact consumer I need to discover attributes about every component used or potentially used in my system. I need consistent practices and patterns for querying for these attributes across many artifact types.
	* As an artifact consumer, I need to discover attributes about every component used or potentially used in my system. I need consistent practices and patterns for querying for these attributes across many artifact types.

[sabre1041]

Suggested change

	* As an artifact provider I need to deliver complete components to a runtime platform which may be completely disconnected from the environment where those components are developed. I want consistent practices for bundling and deploying different component types.
	* As an artifact provider, I need to deliver a complete set of components to a runtime platform which may be fully disconnected from the environment from where those components are developed or assembled. I want consistent practices for bundling and deploying different component types.

[sabre1041]

Suggested change

	* As a security analyst I need to understand and verify the content of bundles used in software and deployed on my platforms.
	* As a security analyst, I need to understand and verify the content of bundles used in software and deployed on my platforms.

[sabre1041]

Suggested change

	* [Proposed] As a platform operator I need a consistent pattern and implementation for deploying bundles to provide additional platform capabilities. For example, I need a consistent pattern for adding extension controller-managers (operators) to Kubernetes.
	* [Proposed] As a platform operator, I need a consistent pattern and implementation approach for deploying bundles to provide additional platform capabilities. For example, I need a defined pattern for adding extension controller-managers (operators) to Kubernetes.

[joshgav]

I'm fine with these changes or the original. @sabre1041 in meetings we agreed that we're going to iterate on these use cases as one of the first tasks of the established WG.

[sabre1041]

Suggested change

In the distributed world of cloud-native computing different artifacts and packages are used to transport configuration and code for the many services and capabilities that comprise and support workloads and applications. As an example, today ArtifactHub advertises that it can crawl about 20 such artifact types [1]. These "cloud-native" bundles add to the previous proliferation of package and artifact types for software dependencies such as Maven and npm and for system packages like deb and rpm.

In the distributed world of cloud-native computing, different artifacts and packages are used to transport configuration and code for the many services and capabilities that comprise and support various types of workloads. As an example, today ArtifactHub advertises that it can crawl about 20 such artifact types [1]. These "cloud-native" bundles adds to the previous proliferation of package and artifact types for software dependencies, such as Maven and npm and for system packages, like deb and rpm.

[sabre1041]

Suggested change

	1. Cloud-native artifact types not currently packaged in OCI, e.g. those in ArtifactHub [3]
	1. Cloud-native artifact types not currently packaged in OCI format, e.g. those in ArtifactHub [3]

[sabre1041]

Suggested change

	2. Software library artifacts such as those in npm, pyPi and Maven Central
	2. Software library artifacts, such as those in npm, PyPi, and Maven Central

[sabre1041]

Suggested change

	3. System package artifacts such as those in rpm, deb, brew
	3. System package artifacts, such as those in RPM, deb, and brew repositories

[sabre1041]

Suggested change

	* Advocate for and contribute to common formats to enable search and discovery of attributes and aspects of artifacts like SBOMs, attestations and other elements
	* Advocate for and contribute to common formats to enable search and discovery of attributes and aspects of artifacts like SBOM's, attestations, and other elements

[sabre1041]

Suggested change

	* Demonstrate via prototypes use of published schemas to facilitate query and analysis of bundles and content.
	* Demonstrate via prototypes the use of published schemas to facilitate query and analysis of bundles and content.

[SteveLasker]

In the spirit of "Perfect is the enemy of good", I approve to continue making iterative progress.

[justincormack]

LGTM

[joshgav]

Really like how you refactored the Goals section @afflom. You captured the intent of the original list items but described them more clearly in human terms.

LGTM!