Proposal: TPR to beta proposal

contributors/design-proposals/thirdpartyresources.md

@@ -0,0 +1,253 @@
+# Moving ThirdPartyResources to beta
+
+## Background
+There are a number of important issues with the alpha version of
+ThirdPartyResources that we wish to address to move TPR to beta. The list is
+tracked [here](https://github.com/kubernetes/features/issues/95), and also
+includes feedback from existing Kubernetes ThirdPartyResource users. This
+proposal covers the steps we believe are necessary to move TPR to beta and to
+prevent future challenges in upgrading.
+
+
+## Goals
+1. Ensure ThirdPartyResource APIs operate consistently with first party
+Kubernetes APIs.
+2. Enable ThirdPartyResources to specify how they will appear in API
+discovery to be consistent with other resources and avoid naming confilcts
+3. Move TPR into their own API group to allow the extensions group to be
+[removed](https://github.com/kubernetes/kubernetes/issues/43214)
+4. Support cluster scoped TPR resources
+5. Identify other features required for TPR to become beta
+6. Minimize the impact to alpha ThirdPartyResources consumers and define a
+process for how TPR migrations / breaking changes can be accomplished (for
+both the cluster and for end users)
+
+Non-goals
+1. Solve automatic conversion of TPR between versions or automatic migration of
+existing TPR
+
+### Desired API Semantics
+TPRs are intended to look like normal kube-like resources to external clients.
+In order to do that effectively, they should respect the normal get, list,
+watch, create, patch, update, and delete semantics.
+
+In "normal" Kubernetes APIs, if I have a persisted resource in the same group
+with the same name in v1 and v2, they are backed by the same underlying object.
+A change made to one is reflected in the other. API clients, garbage collection,
+namespace cleanup, version negotiation, and controllers all build on this.
+
+The convertibility of Kubernetes APIs provides a seamless interaction between
+versions.  A TPR does not have the ability to convert between versions, which
+focuses on the primary role of TPR as an easily extensible and simple mechanism
+for adding new APIs. Conversion primarily allows structural, but not backwards
+incompatible, changes. By not supporting conversion, all TPR use cases are
+preserved, but a large amount of complexity is avoided for consumers of TPR.
+
+Allowing a single, user specified version for a given TPR will provide this
+semantic by preventing server-side versioning altogether.  All instances of a
+single TPR must have the same version or the Kubernetes API semantic of always
+returning a resource encoded to the matching version will not be maintained.
+Since conversions (even native Kubernetes conversions) cannot be used to handle
+behavioral changes, the same effect can be achieved for TPRs client-side with
+overlapping serialization changes.
+
+
+### Avoiding Naming Problems
+There are several identifiers that a Kubernetes API resource has which share
+value-spaces within an API group and must not conflict.  They are:
+1. Resource-type value space
+  1. plural resource-type name - like "configmaps"
+  2. singular resource-type name - like "configmap"
+  3. short names - like "cm"
+2. Kind-type value space - for group "example.com"
+  1. Kind name - like "ConfigMap"
+  2. ListKind name - like "ConfigMapList"
+If these values conflict within their value-spaces then no client will be able
+to properly distinguish intent.
+
+The actual name of the TPR-registration (resource that describes the TPR to
+create) resource can only protect one of these values from conflict.  Since
+Kubernetes API types are accessed via a URL that looks like `/apis/<group>/<version>/namespaces/<namespace-name>/<plural-resource-type>`,
+the name of the TPR-registration object will be `<plural-resource-type>.<group>`.
+
+Conflicts with other parts of the value-space can not be detected with static
+validation, so there will be a spec/status split with `status.conditions` that
+reflect the acceptance status of a TPR-registration.  For instance, you cannot
+determine whether two TPRs in the same group have the same short name without
+inspecting the current state of existing TPRs.
+
+Parts of the value-space will be "claimed" by making an entry in TPR.status to
+include the accepted names which will be served.  This prevents a new TPR from
+disabling an existing TPR's name.
+
+
+## New API
+In order to: 
+1. eliminate opaquely derived information - deriving camel-cased kind names
+from lower-case dash-delimited values as for instance.
+1. allow the expression of complex transformations - not all plurals are easily
+determined (ox and oxen) and not all are English.  Fields for complete
+specification eliminates ambiguity.
+1. handle TPR-registration value-space conflicts
+1. [stop using the extensions API group](https://github.com/kubernetes/kubernetes/issues/43214)
+
+We can create a type `ThirdPartyResource.apiextension.k8s.io`.
+```go
+// ThirdPartyResourceSpec describe how a user wants their resource to appear
+type ThirdPartyResourceSpec struct {
+	// Group is the group this resource belongs in
+	Group string `json:"group" protobuf:"bytes,1,opt,name=group"`
+	// Version is the version this resource belongs in
+	Version string `json:"version" protobuf:"bytes,2,opt,name=version"`
+	// Names holds the information about the resource and kind you have chosen which is
+	// surfaced through discovery.
+	Names ThirdPartyResourceNames
+
+	// Scope indicates whether this resource is cluster or namespace scoped.  Default is namespaced
+	Scope ResourceScope `json:"scope" protobuf:"bytes,8,opt,name=scope,casttype=ResourceScope"`
+}
+
+type ThirdPartyResourceNames struct {
+	// Plural is the plural name of the resource to serve.  It must match the name of the TPR-registration
+	// too: plural.group
+	Plural string `json:"plural" protobuf:"bytes,3,opt,name=plural"`
+	// Singular is the singular name of the resource.  Defaults to lowercased <kind>
+	Singular string `json:"singular,omitempty" protobuf:"bytes,4,opt,name=singular"`
+	// ShortNames are short names for the resource.
+	ShortNames []string `json:"shortNames,omitempty" protobuf:"bytes,5,opt,name=shortNames"`
+	// Kind is the serialized kind of the resource
+	Kind string `json:"kind" protobuf:"bytes,6,opt,name=kind"`
+	// ListKind is the serialized kind of the list for this resource.  Defaults to <kind>List
+	ListKind string `json:"listKind,omitempty" protobuf:"bytes,7,opt,name=listKind"`
+}
+
+type ResourceScope string
+
+const (
+	ClusterScoped    ResourceScope = "Cluster"
+	NamespaceScoped  ResourceScope = "Namespaced"
+)
+
+type ConditionStatus string
+
+// These are valid condition statuses. "ConditionTrue" means a resource is in the condition.
+// "ConditionFalse" means a resource is not in the condition. "ConditionUnknown" means kubernetes
+// can't decide if a resource is in the condition or not. In the future, we could add other
+// intermediate conditions, e.g. ConditionDegraded.
+const (
+	ConditionTrue    ConditionStatus = "True"
+	ConditionFalse   ConditionStatus = "False"
+	ConditionUnknown ConditionStatus = "Unknown"
+)
+
+// ThirdPartyResourceConditionType is a valid value for ThirdPartyResourceCondition.Type
+type ThirdPartyResourceConditionType string
+
+const (
+	// NameConflict means the resource or kind names chosen for this ThirdPartyResource conflict with others in the group.
+	// The first TPR in the group to have the name reflected in status "wins" the name.
+	NameConflict ThirdPartyResourceConditionType = "NameConflict"
+	// Terminating means that the ThirdPartyResource has been deleted and is cleaning up.
+	Terminating ThirdPartyResourceConditionType = "Terminating"
+)
+
+// ThirdPartyResourceCondition contains details for the current condition of this ThirdPartyResource.
+type ThirdPartyResourceCondition struct {
+	// Type is the type of the condition.
+	Type ThirdPartyResourceConditionType `json:"type" protobuf:"bytes,1,opt,name=type,casttype=ThirdPartyResourceConditionType"`
+	// Status is the status of the condition.
+	// Can be True, False, Unknown.
+	Status ConditionStatus `json:"status" protobuf:"bytes,2,opt,name=status,casttype=ConditionStatus"`
+	// Last time the condition transitioned from one status to another.
+	// +optional
+	LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty" protobuf:"bytes,4,opt,name=lastTransitionTime"`
+	// Unique, one-word, CamelCase reason for the condition's last transition.
+	// +optional
+	Reason string `json:"reason,omitempty" protobuf:"bytes,5,opt,name=reason"`
+	// Human-readable message indicating details about last transition.
+	// +optional
+	Message string `json:"message,omitempty" protobuf:"bytes,6,opt,name=message"`
+}
+
+// ThirdPartyResourceStatus indicates the state of the ThirdPartyResource
+type ThirdPartyResourceStatus struct {
+	// Conditions indicate state for particular aspects of a ThirdPartyResource
+	Conditions []ThirdPartyResourceCondition `json:"conditions" protobuf:"bytes,1,opt,name=conditions"`
+
+	// AcceptedNames are the names that are actually being used to serve discovery
+	// They may not be the same as names in spec.
+	AcceptedNames ThirdPartyResourceNames
+}
+
+// +genclient=true
+
+// ThirdPartyResource represents a resource that should be exposed on the API server.  Its name MUST be in the format
+// <.spec.plural>.<.spec.group>.
+type ThirdPartyResource struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
+
+	// Spec describes how the user wants the resources to appear
+	Spec ThirdPartyResourceSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
+	// Status indicates the actual state of the ThirdPartyResource
+	Status ThirdPartyResourceStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
+}
+
+// ThirdPartyResourceList is a list of ThirdPartyResource objects.
+type ThirdPartyResourceList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
+
+	// Items individual ThirdParties
+	Items []ThirdPartyResource `json:"items" protobuf:"bytes,2,rep,name=items"`
+}
+```
+
+
+## Behavior
+### Create
+When a new TPR is created, no synchronous action is taken.
+A controller will run to confirm that value-space of the reserved names doesn't
+collide and sets the "KindNameConflict" condition to `false`.
+
+A custom `http.Handler` will look at request and use the parsed out
+GroupVersionResource information to match it to a ThirdPartyResource.  The ThirdPartyResource
+will be checked to make sure its valid enough in .Status to serve and will
+response appropriated.  If there is no ThirdPartyResource defined, it will delegate
+to the next handler in the chain.
+
+### Delete
+When a TPR-registration is deleted, it will be handled as a finalizer like a
+namespace is done today.  The `Terminating` condition will be updated (like
+namespaces) and that will cause mutating requests to be rejected by the REST
+handler (see above).  The finalizer will remove all the associated storage.
+Once the finalizer is done, it will delete the TPR-registration itself.
+
+
+## Migration from existing TPR
+Because of the changes required to meet the goals, there is not a silent
+auto-migration from the existing TPR to the new TPR.  It will be possible, but
+it will be manual.  At a high level, you simply:
+ 1. Stop all clients from writing to TPR (revoke edit rights for all users) and
+ stop controllers.
+ 2. Get all your TPR-data.  
+ `$ kubectl get TPR --all-namespaces -o yaml > data.yaml`
+ 3. Delete the old TPR-data.  Be sure you orphan!  
+ `$ kubectl delete TPR --all --all-namespaces --cascade=false`
+ 4. Delete the old TPR-registration.  
+ `$ kubectl delete TPR/name`
+ 5. Create a new TPR-registration with the same GroupVersionKind as before.  
+ `$ kubectl create -f new_tpr.name`
+ 6. Recreate your new TPR-data.  
+ `$ kubectl create -f data.yaml`
+ 7. Restart controllers.
+
+There are a couple things that you'll need to consider:
+ 1. Garbage collection.  You may have created links that weren't respected by
+ the GC collector in 1.6.  Since you orphaned your dependents, you'll probably
+ want to re-adopt them like the Kubernetes controllers do with their resources.
+ 2. Controllers will observe deletes.  Part of this migration actually deletes
+ the resource.  Your controller will see the delete.  You ought to shut down
+ your TPR controller while you migrate your data.  If you do this, your
+ controller will never see a delete.
+