Generate a Cluster Service Version (CSV) from an Operator SDK project

[estroz]

A CSV allows an operator to be managed by the Operator Lifecycle Manager. The SDK should generate a CSV using manifests and code available in an operator project structurally similar to that generated by the SDK.

A design doc exists which contains design and implementation details.

This issue should be the source of truth for requests and suggestions regarding the CSV generation process.

Initial implementation: #673

[estroz]

@aravindhp @djzager I'm moving the discussion from #673 here.

@aravindhp If we're going to add marketplace annotations by default, that change will go in another PR.

@djzager as per your comment, you'd rather have the user add marketplace annotations than the SDK?

[djzager]

I'm moving the discussion from #673 here.

Excellent idea. I feel awful for hopping onto your PR the way that I did. Just felt it was easier to make sure we were meeting in the middle before the PR was accepted (and discuss any potential future changes) than after.

It may help you to know where I come from. I've been dabbling around in operator-sdk, specifically around getting the Ansible based Operators working well and then playing around with creating Ansible based Operators like this one. My actual work around Operators has been to get the Ansible Service Broker and Template Service Broker operators ready for 4.0. I want to do what I can to help make sure that operator-sdk makes it easy for users to both create their Operator (Helm, Go, Ansible) as well as make it available via Marketplace and OLM.

as per your comment, you'd rather have the user add marketplace annotations than the SDK?

I view the SDK managing the marketplace annotations as a "nice to have". If the story for users is that they use operator-sdk to make their Operator and then generate the CSV, and there are instructions from Marketplace on specific annotations to add then I would find that workflow straightforward and I personally would have no complaints. Just don't clobber the annotations when I try to use operator-sdk to update the CSV 😄

I would like to try and add a little more detail to my original comment. While I am not wholly aware of the history of Operators + OLM and where CSVs were managed (please correct me if I am wrong), but this feels like a turning point where each project that wants to integrate with Marketplace (OLM via proxy) needs to manage their own CSVs. I suspect that as we move forward in time and, for example, I need to be able to test upgrade scenarios from one version of my Operator to another and I will want to have my older CSVs around to assist in testing and development. Knowing that operator-sdk is opinionated (and rightly so), I think now is the time to have the support for multiple versions in the olm-catalog directory. Something like:

deploy
└── olm-catalog
    ├── 0.1.0
    │   ├── helloworldoperator.0.1.0.clusterserviceversion.yaml
    │   ├── helloworld.crd.yaml
    ├── 0.2.0
    │   ├── helloworldoperator.0.2.0.clusterserviceversion.yaml
    │   ├── helloworld.crd.yaml
    └── helloworld.package.yaml

In practice I may only include the latest CSV in whatever payload I hand off to Marketplace but having 1) the standard layout for the olm-catalog and 2) the flexibility to give all or some of the CSVs in that payload strikes me as desirable.

Hope this helps.

[estroz]

@djzager not a problem! Feel free to jump in on any PR you want. I figured it would be easier to track CSV discussion in an issue rather than a PR that (hopefully) will be merged soon. Everything discussed here will likely be in another PR, especially version management.

The SDK team had a discussion a while ago about how the SDK would handle CSV file naming and versioning. We should rekindle the discussion here and now. There are arguments for two different versioning styles (listed in this comment in a previous PR):

1. The CSV will always be at deploy/olm-catalog/operatorname.csv.yaml.
   • Users should rely on the spec.replaces field, as does the OLM, and git for versioning.
   • This is how Implementation of CSV scaffold and update process #673 assumes CSV's are managed currently.
2. The CSV file name (or directory, or both) will include the CSV version.
   • Users can version-control their CSV's as they see fit.

I've seen at least one project manage CSV's in a scheme similar to the one you're suggesting (2), and I'm sure others do as well. I'd like to get to understand the advantages of doing so; in my (uninformed) opinion, spec.replaces in combination with git tags handles versioning just fine and I think the SDK should be similarly opinionated. The OLM is able to handle (1) according to @ecordell's comment. I'm definitely open to (2) if the argument is strong.

@hasbro17 @spahl @ecordell thoughts?

[djzager]

1. The CSV will always be at deploy/olm-catalog/operatorname.csv.yaml.

• Users should rely on the spec.replaces field, as does the OLM, and git for versioning.
• This is how Implementation of CSV scaffold and update process #673 assumes CSV's are managed currently.

2. The CSV file name (or directory, or both) will include the CSV version.

• Users can version-control their CSV's as they see fit.

I'll do my best to articulate why I think method (2) is the correct answer. Right now, the two Operators that I am directly involved in only have one version 0.1.0 of the CSV. It won't be long before we want to create new versions of this CSV. When we go through the effort to create a new CSV version (for this example 0.2.0) we will want to have an upgrade path from 0.1.0 --> 0.2.0 and at some point we will want to test this upgrade path. I would expect this CSV upgrade testing to be done as part of the development effort for the 0.2.0 version of the CSV. In order to perform this test I will need to load both versions of the CSV into OLM. Jumping through some collection of git checkout or curl $path_to_raw_version_of_old_csv is just not practical when compared to having all versions of the CSV that my operator supports with me in my current view of the project.

[eriknelson]

Jumping through some collection of git checkout or curl $path_to_raw_version_of_old_csv is just not practical

I have to echo this concern. Having to manually git checkout each version I'm interested in is not a great experience.

[estroz]

@djzager @eriknelson Testing is really important to us, so unless the OLM has tool to manage CSV's for testing in this manner, which I don't think it does, I am inclined to implement (2). I had thought using spec.replaces + git would be sufficient, and while that might be true for production, the developer/user experience does suffer. I don't see an easier way than using git to check out CSV versions under (1) as mentioned.

[djzager]

@estroz Awesome. Thank you for tracking this as well as taking the steps necessary to make Operator integration easier.

[hasbro17]

Jumping in a little late but I'm also in agreement with the above discussion that the SDK should generate a name versioned CSV file for each version e.g <operator-name>-0.1.0.csv rather than overwriting the same file for each subsequent version.

[estroz]

@dmesser posted the following in #990:

Currently we aim to generate a ClusterServiceVersion and the CRDs currently found in an operator-sdk project to form a bundle.

For easy local testing using OLM/operator-registry we should also generate manifests to form a package definition. The directory structure should follow <package-name>/<CSV-semantic-version>/ containing one or more CSVs following the proposed naming scheme of <operator-name>.vX.Y.Z.clusterserviceversion.yaml and one or more CRDs following the notation <kind>.crd.yaml.

Example:

manifests
└── etcd
 ├── 0.6.1
 │   ├── etcdcluster.crd.yaml
 │   └── etcdoperator.clusterserviceversion.yaml
 ├── 0.9.0
 │   ├── etcdbackup.crd.yaml
 │   ├── etcdcluster.crd.yaml
 │   ├── etcdoperator.v0.9.0.clusterserviceversion.yaml
 │   └── etcdrestore.crd.yaml
 ├── 0.9.2
 │   ├── etcdbackup.crd.yaml
 │   ├── etcdcluster.crd.yaml
 │   ├── etcdoperator.v0.9.2.clusterserviceversion.yaml
 │   └── etcdrestore.crd.yaml
 └── etcd.package.yaml

This format would also allow to easily commit an operator to the community operator repository. Further alignment between the expected directory structures between these two routes might be necessary.

In the SDK's case, manifests == deploy and etcd == olm-catalog, since an operator cannot have multiple packages. @dmesser is manifests a hard requirement, or can manifests/olm-catalog be used? Writing CSV's to deploy/{operator_name} is redundant imo.

We do not generate an {operator_name}.package.yaml file yet but can add a task to do so.

[estroz]

Another issue this brings up is how CRD's are handled by the SDK. Currently we generate one set of CRD's in deploy/crds which get updated to latest by either the user or various commands. With a registry-like structure the CRD's are semantically versioned by dir. I'll need to think on how the SDK can sync CRD and CSV versioning while minimizing breaking changes.

[dmesser]

@estroz Ideally a user would be able to just use the entire deploy or manifests directory and use it with a Dockerfile as outlined in the build instructions operator-registry and at the same time copy it to community-operatars in a fork of https://github.com/operator-framework/community-operators. So the name of the directory itself would not really be that important.
Without this context deploy/{operator_name} would be redundant but we are looking at a larger process here.

[estroz]

@dmesser we could implement a structure like deploy/olm-catalog/{operator_name}. Now manifests == olm-catalog and users can copy the entire olm-catalog dir as the registry would expect, and the other manifests an SDK operator uses won't be copied. I want to keep olm-catalog or something similarly named so operator devs know the dirs' purpose. WDYT?

[dmesser]

/SGTM

[estroz]

@aravindhp @djzager @dmesser @eriknelson now that #1016 is merged the SDK will version CSV's and CRD's as discussed here. I'll close this issue for now, and reopen it if you have followup questions or suggestions on the implementation. If you find any bugs feel free to open a new issue. Thanks for the help!