From f68ad715d9fa3dcb2bd02f13a4c3121f5203c365 Mon Sep 17 00:00:00 2001 From: deads2k Date: Fri, 24 Mar 2017 14:08:56 -0400 Subject: [PATCH] TPR to beta proposal --- .../design-proposals/thirdpartyresources.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 contributors/design-proposals/thirdpartyresources.md diff --git a/contributors/design-proposals/thirdpartyresources.md b/contributors/design-proposals/thirdpartyresources.md new file mode 100644 index 00000000000..8f651d0d2cc --- /dev/null +++ b/contributors/design-proposals/thirdpartyresources.md @@ -0,0 +1,253 @@ +# Moving ThirdPartyResources to beta + +## Background +There are a number of important issues with the alpha version of +ThirdPartyResources that we wish to address to move TPR to beta. The list is +tracked [here](https://github.com/kubernetes/features/issues/95), and also +includes feedback from existing Kubernetes ThirdPartyResource users. This +proposal covers the steps we believe are necessary to move TPR to beta and to +prevent future challenges in upgrading. + + +## Goals +1. Ensure ThirdPartyResource APIs operate consistently with first party +Kubernetes APIs. +2. Enable ThirdPartyResources to specify how they will appear in API +discovery to be consistent with other resources and avoid naming confilcts +3. Move TPR into their own API group to allow the extensions group to be +[removed](https://github.com/kubernetes/kubernetes/issues/43214) +4. Support cluster scoped TPR resources +5. Identify other features required for TPR to become beta +6. Minimize the impact to alpha ThirdPartyResources consumers and define a +process for how TPR migrations / breaking changes can be accomplished (for +both the cluster and for end users) + +Non-goals +1. Solve automatic conversion of TPR between versions or automatic migration of +existing TPR + +### Desired API Semantics +TPRs are intended to look like normal kube-like resources to external clients. +In order to do that effectively, they should respect the normal get, list, +watch, create, patch, update, and delete semantics. + +In "normal" Kubernetes APIs, if I have a persisted resource in the same group +with the same name in v1 and v2, they are backed by the same underlying object. +A change made to one is reflected in the other. API clients, garbage collection, +namespace cleanup, version negotiation, and controllers all build on this. + +The convertibility of Kubernetes APIs provides a seamless interaction between +versions. A TPR does not have the ability to convert between versions, which +focuses on the primary role of TPR as an easily extensible and simple mechanism +for adding new APIs. Conversion primarily allows structural, but not backwards +incompatible, changes. By not supporting conversion, all TPR use cases are +preserved, but a large amount of complexity is avoided for consumers of TPR. + +Allowing a single, user specified version for a given TPR will provide this +semantic by preventing server-side versioning altogether. All instances of a +single TPR must have the same version or the Kubernetes API semantic of always +returning a resource encoded to the matching version will not be maintained. +Since conversions (even native Kubernetes conversions) cannot be used to handle +behavioral changes, the same effect can be achieved for TPRs client-side with +overlapping serialization changes. + + +### Avoiding Naming Problems +There are several identifiers that a Kubernetes API resource has which share +value-spaces within an API group and must not conflict. They are: +1. Resource-type value space + 1. plural resource-type name - like "configmaps" + 2. singular resource-type name - like "configmap" + 3. short names - like "cm" +2. Kind-type value space - for group "example.com" + 1. Kind name - like "ConfigMap" + 2. ListKind name - like "ConfigMapList" +If these values conflict within their value-spaces then no client will be able +to properly distinguish intent. + +The actual name of the TPR-registration (resource that describes the TPR to +create) resource can only protect one of these values from conflict. Since +Kubernetes API types are accessed via a URL that looks like `/apis///namespaces//`, +the name of the TPR-registration object will be `.`. + +Conflicts with other parts of the value-space can not be detected with static +validation, so there will be a spec/status split with `status.conditions` that +reflect the acceptance status of a TPR-registration. For instance, you cannot +determine whether two TPRs in the same group have the same short name without +inspecting the current state of existing TPRs. + +Parts of the value-space will be "claimed" by making an entry in TPR.status to +include the accepted names which will be served. This prevents a new TPR from +disabling an existing TPR's name. + + +## New API +In order to: +1. eliminate opaquely derived information - deriving camel-cased kind names +from lower-case dash-delimited values as for instance. +1. allow the expression of complex transformations - not all plurals are easily +determined (ox and oxen) and not all are English. Fields for complete +specification eliminates ambiguity. +1. handle TPR-registration value-space conflicts +1. [stop using the extensions API group](https://github.com/kubernetes/kubernetes/issues/43214) + +We can create a type `ThirdPartyResource.apiextension.k8s.io`. +```go +// ThirdPartyResourceSpec describe how a user wants their resource to appear +type ThirdPartyResourceSpec struct { + // Group is the group this resource belongs in + Group string `json:"group" protobuf:"bytes,1,opt,name=group"` + // Version is the version this resource belongs in + Version string `json:"version" protobuf:"bytes,2,opt,name=version"` + // Names holds the information about the resource and kind you have chosen which is + // surfaced through discovery. + Names ThirdPartyResourceNames + + // Scope indicates whether this resource is cluster or namespace scoped. Default is namespaced + Scope ResourceScope `json:"scope" protobuf:"bytes,8,opt,name=scope,casttype=ResourceScope"` +} + +type ThirdPartyResourceNames struct { + // Plural is the plural name of the resource to serve. It must match the name of the TPR-registration + // too: plural.group + Plural string `json:"plural" protobuf:"bytes,3,opt,name=plural"` + // Singular is the singular name of the resource. Defaults to lowercased + Singular string `json:"singular,omitempty" protobuf:"bytes,4,opt,name=singular"` + // ShortNames are short names for the resource. + ShortNames []string `json:"shortNames,omitempty" protobuf:"bytes,5,opt,name=shortNames"` + // Kind is the serialized kind of the resource + Kind string `json:"kind" protobuf:"bytes,6,opt,name=kind"` + // ListKind is the serialized kind of the list for this resource. Defaults to List + ListKind string `json:"listKind,omitempty" protobuf:"bytes,7,opt,name=listKind"` +} + +type ResourceScope string + +const ( + ClusterScoped ResourceScope = "Cluster" + NamespaceScoped ResourceScope = "Namespaced" +) + +type ConditionStatus string + +// These are valid condition statuses. "ConditionTrue" means a resource is in the condition. +// "ConditionFalse" means a resource is not in the condition. "ConditionUnknown" means kubernetes +// can't decide if a resource is in the condition or not. In the future, we could add other +// intermediate conditions, e.g. ConditionDegraded. +const ( + ConditionTrue ConditionStatus = "True" + ConditionFalse ConditionStatus = "False" + ConditionUnknown ConditionStatus = "Unknown" +) + +// ThirdPartyResourceConditionType is a valid value for ThirdPartyResourceCondition.Type +type ThirdPartyResourceConditionType string + +const ( + // NameConflict means the resource or kind names chosen for this ThirdPartyResource conflict with others in the group. + // The first TPR in the group to have the name reflected in status "wins" the name. + NameConflict ThirdPartyResourceConditionType = "NameConflict" + // Terminating means that the ThirdPartyResource has been deleted and is cleaning up. + Terminating ThirdPartyResourceConditionType = "Terminating" +) + +// ThirdPartyResourceCondition contains details for the current condition of this ThirdPartyResource. +type ThirdPartyResourceCondition struct { + // Type is the type of the condition. + Type ThirdPartyResourceConditionType `json:"type" protobuf:"bytes,1,opt,name=type,casttype=ThirdPartyResourceConditionType"` + // Status is the status of the condition. + // Can be True, False, Unknown. + Status ConditionStatus `json:"status" protobuf:"bytes,2,opt,name=status,casttype=ConditionStatus"` + // Last time the condition transitioned from one status to another. + // +optional + LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,4,opt,name=lastTransitionTime"` + // Unique, one-word, CamelCase reason for the condition's last transition. + // +optional + Reason string `json:"reason,omitempty" protobuf:"bytes,5,opt,name=reason"` + // Human-readable message indicating details about last transition. + // +optional + Message string `json:"message,omitempty" protobuf:"bytes,6,opt,name=message"` +} + +// ThirdPartyResourceStatus indicates the state of the ThirdPartyResource +type ThirdPartyResourceStatus struct { + // Conditions indicate state for particular aspects of a ThirdPartyResource + Conditions []ThirdPartyResourceCondition `json:"conditions" protobuf:"bytes,1,opt,name=conditions"` + + // AcceptedNames are the names that are actually being used to serve discovery + // They may not be the same as names in spec. + AcceptedNames ThirdPartyResourceNames +} + +// +genclient=true + +// ThirdPartyResource represents a resource that should be exposed on the API server. Its name MUST be in the format +// <.spec.plural>.<.spec.group>. +type ThirdPartyResource struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` + + // Spec describes how the user wants the resources to appear + Spec ThirdPartyResourceSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"` + // Status indicates the actual state of the ThirdPartyResource + Status ThirdPartyResourceStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"` +} + +// ThirdPartyResourceList is a list of ThirdPartyResource objects. +type ThirdPartyResourceList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` + + // Items individual ThirdParties + Items []ThirdPartyResource `json:"items" protobuf:"bytes,2,rep,name=items"` +} +``` + + +## Behavior +### Create +When a new TPR is created, no synchronous action is taken. +A controller will run to confirm that value-space of the reserved names doesn't +collide and sets the "KindNameConflict" condition to `false`. + +A custom `http.Handler` will look at request and use the parsed out +GroupVersionResource information to match it to a ThirdPartyResource. The ThirdPartyResource +will be checked to make sure its valid enough in .Status to serve and will +response appropriated. If there is no ThirdPartyResource defined, it will delegate +to the next handler in the chain. + +### Delete +When a TPR-registration is deleted, it will be handled as a finalizer like a +namespace is done today. The `Terminating` condition will be updated (like +namespaces) and that will cause mutating requests to be rejected by the REST +handler (see above). The finalizer will remove all the associated storage. +Once the finalizer is done, it will delete the TPR-registration itself. + + +## Migration from existing TPR +Because of the changes required to meet the goals, there is not a silent +auto-migration from the existing TPR to the new TPR. It will be possible, but +it will be manual. At a high level, you simply: + 1. Stop all clients from writing to TPR (revoke edit rights for all users) and + stop controllers. + 2. Get all your TPR-data. + `$ kubectl get TPR --all-namespaces -o yaml > data.yaml` + 3. Delete the old TPR-data. Be sure you orphan! + `$ kubectl delete TPR --all --all-namespaces --cascade=false` + 4. Delete the old TPR-registration. + `$ kubectl delete TPR/name` + 5. Create a new TPR-registration with the same GroupVersionKind as before. + `$ kubectl create -f new_tpr.name` + 6. Recreate your new TPR-data. + `$ kubectl create -f data.yaml` + 7. Restart controllers. + +There are a couple things that you'll need to consider: + 1. Garbage collection. You may have created links that weren't respected by + the GC collector in 1.6. Since you orphaned your dependents, you'll probably + want to re-adopt them like the Kubernetes controllers do with their resources. + 2. Controllers will observe deletes. Part of this migration actually deletes + the resource. Your controller will see the delete. You ought to shut down + your TPR controller while you migrate your data. If you do this, your + controller will never see a delete. +