Add custom channels to cluster spec

[chris-h-phillips]

We have several addons in addition to the ones that kops installs for us that we wanted to manage in our own custom channels. This was discussed a bit here

#2399.

This allows you to specify additional custom channels for protokube.

[chrislovecnm]

/ok-to-test

Many people have been discussing this :) Appreciate the contribution. Need to have @justinsb and @robinpercy review the API change! Let me chew on this a bit as I am wondering if we want to allow the channels and bootstrap files as API objects themselves.

Appreciate the contribution, and please be patient on this PR, this is an awesome one, and we have to get it right.

/assign @justinsb @robinpercy

[robinpercy]

This looks great. I don't have any issue with the API change, but I need to look into how addons that are duplicated between channels would be handled. One of the main use cases I'm looking for is the ability to override default addons.

[k8s-github-robot]

@polarbizzle PR needs rebase

[KashifSaadat]

LGTM, won't add the flag on yet to give time for more thorough review.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: polarbizzle
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: justinsb

Assign the PR to them by writing /assign @justinsb in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[chris-h-phillips]

@KashifSaadat Do you think we're good at this point?

[chrislovecnm]

/assign @justinsb

We need Justin to look at this one, I like it but I am wondering if we need a top level object.

[justinsb]

So we don't really want to expose the channels format as part of our API, because it feels like we're very close to being able to replace channels with something more unified with upstream. Can I ask about your use case @polarbizzle ... would you be happy with just a manifest that we would kubectl apply -f, rather than having the channels wrapper?

Otherwise I'm struggling to think how we can migrate off the channels format. Currently it's all internal, and while a handful of people might have figured out how to define their own channels, I'm betting it's not too many, and they'll be very capable of moving to a new format :-)

[jordanjennings]

@justinsb Not to sidetrack this issue, but what's the upcoming alternative you are referring to?

[chris-h-phillips]

We've just been looking for a good way to manage the addons for our cluster with some mechanism. Right now we just manually kubectl apply -f things. We didn't want to write some new way to maintain the lifecycle of addons but to be consistent with what kops was already doing. If there's some upstream addon management capability that will soon be a part of kops then we would embrace that for sure. I'm with @jordanjennings -- being curious about what the alternative you're hinting at is.

[justinsb]

Ah - the Declarative Application Management ( https://docs.google.com/document/d/1cLPGweVEYrVqQvBLJg6sxV-TrE5Rm2MNOBA_cxZP2WU/edit
) work seems to be coming together for management of system addons. So you'd be able to just kubectl apply -f whatever a channel becomes in that world.

Although actually, now that I've said - that same argument applies here. So we can support a channel manifest here, but we can encourage people to use a "bare" manifest today and then have the new manifest tomorrow. We can tell people that channels manifests will continue to work, but I'll try to have a look at making this just work with bare manifests.

I do think we might want something more like this though:

addons:
  - name: foo
    manifest: https://k8s.io/foo
  - name: bar
    manifest: https://k8s.io/bar

Because I think we'll likely support more options going forward. (We don't even need name today). Would you be OK with that (and if the code is a barrier I can merge this an propose a quick PR to move to objects instead)

One super-interesting option would be to expose some of the automagic addons. Maybe that's a good use for name - if you define kube-dns you can override the built-in kube-dns manifest.

[chris-h-phillips]

For now we're having success publishing our addons into a channel in s3 and having it get picked up and applied the same way the bootstrap channel does. I can see how we'd want to improve the addon story as a whole in some of the ways you mention. I'll defer to you on what the most appropriate way forward is.

[so0k]

As discussed in the Issue referenced, we also manage custom channels and this PR would be very helpful for us

At the moment, we apply channels manually using make targets after the cluster is provisioned and on ad-hoc basis.

Our channels use case is mainly through Terraform templated yaml manifests

• Create pre-defined namespaces, rbac and a locked down tiller per namespace
• Create cluster-wide rbac / service accounts
• some upstream kops channels that are not part of bootstrap

rest of the cluster deployments is managed using helm and through CI/CD (using the service accounts created during bootstrap)

I will read up on the Declarative Application Management document to see how we can migrate to that

[justinsb]

So I'm very happy to get this in, I just ask that we tweak the schema a bit to support the richer model we have going forwards. Let me propose a PR based on this one :-)

[justinsb]

Please take a look at #4538 to see if that works for you - it's just switching it to an object to allow for more options in future.

[k8s-ci-robot]

@polarbizzle: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[justinsb]

Merged as part of #4538 - thanks for getting this in - excited to play with this :-)