-
Notifications
You must be signed in to change notification settings - Fork 5.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Proposal: SubResources for CustomResources
- Loading branch information
Showing
1 changed file
with
221 additions
and
0 deletions.
There are no files selected for viewing
221 changes: 221 additions & 0 deletions
221
contributors/design-proposals/customresources-subresources.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,221 @@ | ||
| # Subresources for CustomResources | ||
|
|
||
| Authors: @nikhita, @sttts | ||
|
|
||
| ## Table of Contents | ||
|
|
||
| 1. [Abstract](#abstract) | ||
| 2. [Goals](#goals) | ||
| 3. [Non-Goals](#non-goals) | ||
| 4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition) | ||
| 1. [API Types](#api-types) | ||
| 2. [Feature Gate](#feature-gate) | ||
| 5. [Semantics](#semantics) | ||
| 1. [Validation Behavior](#validation-behavior) | ||
| 1. [Status](#status) | ||
| 2. [Scale](#scale) | ||
| 2. [Status Behavior](#status-behavior) | ||
| 3. [Scale Behavior](#scale-behavior) | ||
| 1. [Status Replicas Behavior](#status-replicas-behavior) | ||
| 2. [Selector Behavior](#selector-behavior) | ||
| 4. [Implementation Plan](#implementation-plan) | ||
| 5. [Alternatives](#alternatives) | ||
| 1. [Scope](#scope) | ||
|
|
||
| ## Abstract | ||
|
|
||
| [CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs. | ||
|
|
||
| However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources. | ||
|
|
||
| ## Goals | ||
|
|
||
| 1. Support status/spec split for CustomResources: | ||
| 1. Status changes are ignored on the main resource endpoint. | ||
| 2. Support a `/status` subresource HTTP path for status changes. | ||
| 3. `metadata.Generation` is increased only on spec changes. | ||
| 2. Support a `/scale` subresource for CustomResources. | ||
| 3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources. | ||
| 4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint. | ||
| 5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708). | ||
|
|
||
| ## Non-Goals | ||
|
|
||
| 1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`. | ||
| 2. Unify the many `Scale` types: as long as there is no generic `Scale` object in Kubernetes, we will propose to introduce yet another `Scale` type in the `apiextensions.k8s.io` api group. If Kubernetes switches to a generic `Scale` object, `apiextensions.k8s.io` will follow. | ||
|
|
||
| ## Proposed Extension of CustomResourceDefinition | ||
|
|
||
| ### API Types | ||
|
|
||
| The addition of the following external types in `apiextensions.k8s.io/v1beta1` is proposed: | ||
|
|
||
| ```go | ||
| type CustomResourceDefinitionSpec struct { | ||
| ... | ||
| // SubResources describes the subresources for CustomResources | ||
| // This field is alpha-level and should only be sent to servers that enable | ||
| // subresources via the CurstomResourceSubResources feature gate. | ||
| // +optional | ||
| SubResources *CustomResourceSubResources `json:“subResources,omitempty”` | ||
| } | ||
|
|
||
| // CustomResourceSubResources defines the status and scale subresources for CustomResources. | ||
| type CustomResourceSubResources struct { | ||
| // Status denotes the status subresource for CustomResources | ||
| Status *CustomResourceSubResourceStatus `json:“status,omitempty”` | ||
| // Scale denotes the scale subresource for CustomResources | ||
| Scale *CustomResourceSubResourceScale `json:“scale,omitempty”` | ||
| } | ||
|
|
||
| // CustomResourceSubResourceStatus defines how to serve the HTTP path <CR Name>/status. | ||
| type CustomResourceSubResourceStatus struct { | ||
| // The JSON path (e.g. “.status”) of the status of a CustomResource. | ||
| // The whole object is transferred over the wire, but only the status can be mutated via the /status subresource. | ||
| // StatusPath is restricted to be “.status” and defaults to “.status”. | ||
| StatusPath string `json:“statusPath,omitempty”` | ||
| // The JSON path (e.g. “.spec”) of the spec of a CustomResource. | ||
| // SpecPath is restricted to be “.spec” and defaults to “.spec”. | ||
| // Changes to the specified JSON path increase the “.metadata.generation” value. | ||
| SpecPath string `json:“specPath,omitempty”` | ||
| } | ||
|
|
||
| // CustomResourceSubResourceScale defines how to serve the HTTP path <CR name>/scale. | ||
| type CustomResourceSubResourceScale struct { | ||
| // required, e.g. “.spec.replicas”. | ||
| // Only JSON paths without the array notation are allowed. | ||
| SpecReplicasPath string `json:“specReplicasPath,omitempty”` | ||
| // optional, e.g. “.status.replicas”. | ||
| // Only JSON paths without the array notation are allowed. | ||
| StatusReplicasPath string `json:“statusReplicasPath,omitempty”` | ||
| // optional, e.g. “.spec.labelSelector”. | ||
| // Only JSON paths without the array notation are allowed. | ||
| LabelSelectorPath string `json:“labelSelectorPath,omitempty”` | ||
| } | ||
|
|
||
| // The following is the payload to send over the wire for /scale. It happens | ||
| // to be defined here in apiextensions.k8s.io because we don’t have a global | ||
| // Scale type in meta/v1. Ref: https://github.com/kubernetes/kubernetes/issues/49504 | ||
|
|
||
| // Scale represents a scaling request for a resource. | ||
| type Scale struct { | ||
| metav1.TypeMeta `json:",inline"` | ||
| // Standard object metadata; More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata. | ||
| // +optional | ||
| metav1.ObjectMeta `json:"metadata,omitempty"` | ||
|
|
||
| // defines the behavior of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. | ||
| // +optional | ||
| Spec ScaleSpec `json:"spec,omitempty"` | ||
|
|
||
| // current status of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. Read-only. | ||
| // +optional | ||
| Status ScaleStatus `json:"status,omitempty"` | ||
| } | ||
|
|
||
| // ScaleSpec describes the attributes of a scale subresource. | ||
| type ScaleSpec struct { | ||
| // desired number of instances for the scaled object. | ||
| // +optional | ||
| Replicas int32 `json:"replicas,omitempty"` | ||
| } | ||
|
|
||
| // ScaleStatus represents the current status of a scale subresource. | ||
| type ScaleStatus struct { | ||
| // actual number of observed instances of the scaled object. | ||
| Replicas int32 `json:"replicas"` | ||
|
|
||
| // label query that should match the replicas count. This is same | ||
| // as the label selector but in the string format to avoid introspection | ||
| // by clients. The string will be in the same format as the query-param syntax. | ||
| // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors | ||
| // +optional | ||
| Selector string `json:"selector"` | ||
| } | ||
| ``` | ||
|
|
||
| ### Feature Gate | ||
|
|
||
| The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate. | ||
| If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is dropped on creation and updates of CRDs. | ||
|
|
||
| ## Semantics | ||
|
|
||
| ### Validation Behavior | ||
|
|
||
| #### Status | ||
|
|
||
| The status endpoint of a CustomResource receives a full CR object. Changes outside of the status path are ignored. | ||
| For validation everything outside of the JSON Path `StatusPath` can be reset to the existing values of the CustomResource. Then the JSON Schema presented in the CRD is validated against the whole object. | ||
|
|
||
| Note: one could extract the status subobject, filter the JSON schema by rules applied to the status and then validate the status subobject only. | ||
| The filter step of the JSON schema though is not obvious in the case that there are status validations not below a properties JSON Schema construct. For that reason, the alternative implementation is preferred as its semantics are much simpler. | ||
|
|
||
| #### Scale | ||
|
|
||
| Moreover, if the scale subresource is enabled: | ||
|
|
||
| - The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is validated to ensure that it contains non-negative integer value and is not empty. | ||
|
|
||
| - The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is validated to be a an integer number if it exists (i.e. this can be empty). | ||
|
|
||
| - The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is validated to be a valid label selector if it exists (i.e. this can be empty). | ||
| If it is invalid, an empty `LabelSelector` is used. | ||
|
|
||
| ### Status Behavior | ||
|
|
||
| If the `/status` subresource is enabled, the following behaviors change: | ||
|
|
||
| - The main resource endpoint will ignore all changes in the specified status subpath. | ||
| (note: it will **not** reject requests which try to change the status, following the existing semantics of other resources). | ||
|
|
||
| - The `.metadata.generation` field is updated if and only if the value at the specified `SpecPath` (e.g. `.spec`) changes. | ||
| Additionally, if the spec does not change, `.metadata.generation` is not updated. | ||
|
|
||
| - The `/status` subresource receives a full resource object, but only considers the value at the specified `StatusPath` subpath (e.g. `.status`) for the update. | ||
| The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539. | ||
|
|
||
| Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer. | ||
|
|
||
| ### Scale Behavior | ||
|
|
||
| The number of CustomResources can be easily scaled up or down depending on the replicas field present in path specified by `SpecPath`. | ||
|
|
||
| Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified: | ||
|
|
||
| 1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`. | ||
|
|
||
| 2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`. | ||
| In other words, the scale and the CustomResource share the resource version used for optimistic concurrency. | ||
| Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource. | ||
|
|
||
| #### Status Replicas Behavior | ||
|
|
||
| As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only. | ||
|
|
||
| #### Selector Behavior | ||
|
|
||
| `CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count. | ||
| The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty. | ||
| Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources. | ||
|
|
||
| ## Implementation Plan | ||
|
|
||
| The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much): | ||
|
|
||
| 1. `/status` subresource | ||
| 2. `/scale` subresource | ||
|
|
||
| ## Alternatives | ||
|
|
||
| ### Scope | ||
|
|
||
| In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`. | ||
| We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`. | ||
| Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions. | ||
|
|
||
| **Note**: that the types do not make the addition of other subresources impossible in the future. | ||
|
|
||
| We also restrict the JSON path for the status and the spec within the CustomResource. | ||
| We could make them definable by the user and the proposed types actually allow us to open this up in the future. | ||
| For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well. |