From ff5d1de77fb8dce19167cd40bc2b1239fcb493a0 Mon Sep 17 00:00:00 2001 From: Nikhita Raghunath Date: Tue, 15 Aug 2017 20:17:33 +0530 Subject: [PATCH] Proposal: SubResources for CustomResources --- .../customresources-subresources.md | 221 ++++++++++++++++++ 1 file changed, 221 insertions(+) create mode 100644 contributors/design-proposals/customresources-subresources.md diff --git a/contributors/design-proposals/customresources-subresources.md b/contributors/design-proposals/customresources-subresources.md new file mode 100644 index 00000000000..db5e135bbfc --- /dev/null +++ b/contributors/design-proposals/customresources-subresources.md @@ -0,0 +1,221 @@ +# Subresources for CustomResources + +Authors: @nikhita, @sttts + +## Table of Contents + +1. [Abstract](#abstract) +2. [Goals](#goals) +3. [Non-Goals](#non-goals) +4. [Proposed Extension of CustomResourceDefinition](#proposed-extension-of-customresourcedefinition) + 1. [API Types](#api-types) + 2. [Feature Gate](#feature-gate) +5. [Semantics](#semantics) + 1. [Validation Behavior](#validation-behavior) + 1. [Status](#status) + 2. [Scale](#scale) + 2. [Status Behavior](#status-behavior) + 3. [Scale Behavior](#scale-behavior) + 1. [Status Replicas Behavior](#status-replicas-behavior) + 2. [Selector Behavior](#selector-behavior) +4. [Implementation Plan](#implementation-plan) +5. [Alternatives](#alternatives) + 1. [Scope](#scope) + +## Abstract + +[CustomResourceDefinitions](https://github.com/kubernetes/community/pull/524) (CRDs) were introduced in 1.7. The objects defined by CRDs are called CustomResources (CRs). Currently, we do not provide subresources for CRs. + +However, it is one of the [most requested features](https://github.com/kubernetes/kubernetes/issues/38113) and this proposal seeks to add `/status` and `/scale` subresources for CustomResources. + +## Goals + +1. Support status/spec split for CustomResources: + 1. Status changes are ignored on the main resource endpoint. + 2. Support a `/status` subresource HTTP path for status changes. + 3. `metadata.Generation` is increased only on spec changes. +2. Support a `/scale` subresource for CustomResources. +3. Maintain backward compatibility by allowing CRDs to opt-in to enable subresources. +4. If a CustomResource is already structured using spec/status, allow it to easily transition to use the `/status` and `/scale` endpoint. +5. Work seamlessly with [JSON Schema validation](https://github.com/kubernetes/community/pull/708). + +## Non-Goals + +1. Allow defining arbitrary subresources i.e. subresources except `/status` and `/scale`. +2. Unify the many `Scale` types: as long as there is no generic `Scale` object in Kubernetes, we will propose to introduce yet another `Scale` type in the `apiextensions.k8s.io` api group. If Kubernetes switches to a generic `Scale` object, `apiextensions.k8s.io` will follow. + +## Proposed Extension of CustomResourceDefinition + +### API Types + +The addition of the following external types in `apiextensions.k8s.io/v1beta1` is proposed: + +```go +type CustomResourceDefinitionSpec struct { + ... + // SubResources describes the subresources for CustomResources + // This field is alpha-level and should only be sent to servers that enable + // subresources via the CurstomResourceSubResources feature gate. + // +optional + SubResources *CustomResourceSubResources `json:“subResources,omitempty”` +} + +// CustomResourceSubResources defines the status and scale subresources for CustomResources. +type CustomResourceSubResources struct { + // Status denotes the status subresource for CustomResources + Status *CustomResourceSubResourceStatus `json:“status,omitempty”` + // Scale denotes the scale subresource for CustomResources + Scale *CustomResourceSubResourceScale `json:“scale,omitempty”` +} + +// CustomResourceSubResourceStatus defines how to serve the HTTP path /status. +type CustomResourceSubResourceStatus struct { + // The JSON path (e.g. “.status”) of the status of a CustomResource. + // The whole object is transferred over the wire, but only the status can be mutated via the /status subresource. + // StatusPath is restricted to be “.status” and defaults to “.status”. + StatusPath string `json:“statusPath,omitempty”` + // The JSON path (e.g. “.spec”) of the spec of a CustomResource. + // SpecPath is restricted to be “.spec” and defaults to “.spec”. + // Changes to the specified JSON path increase the “.metadata.generation” value. + SpecPath string `json:“specPath,omitempty”` +} + +// CustomResourceSubResourceScale defines how to serve the HTTP path /scale. +type CustomResourceSubResourceScale struct { + // required, e.g. “.spec.replicas”. + // Only JSON paths without the array notation are allowed. + SpecReplicasPath string `json:“specReplicasPath,omitempty”` + // optional, e.g. “.status.replicas”. + // Only JSON paths without the array notation are allowed. + StatusReplicasPath string `json:“statusReplicasPath,omitempty”` + // optional, e.g. “.spec.labelSelector”. + // Only JSON paths without the array notation are allowed. + LabelSelectorPath string `json:“labelSelectorPath,omitempty”` +} + +// The following is the payload to send over the wire for /scale. It happens +// to be defined here in apiextensions.k8s.io because we don’t have a global +// Scale type in meta/v1. Ref: https://github.com/kubernetes/kubernetes/issues/49504 + +// Scale represents a scaling request for a resource. +type Scale struct { + metav1.TypeMeta `json:",inline"` + // Standard object metadata; More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata. + // +optional + metav1.ObjectMeta `json:"metadata,omitempty"` + + // defines the behavior of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. + // +optional + Spec ScaleSpec `json:"spec,omitempty"` + + // current status of the scale. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#spec-and-status. Read-only. + // +optional + Status ScaleStatus `json:"status,omitempty"` +} + +// ScaleSpec describes the attributes of a scale subresource. +type ScaleSpec struct { + // desired number of instances for the scaled object. + // +optional + Replicas int32 `json:"replicas,omitempty"` +} + +// ScaleStatus represents the current status of a scale subresource. +type ScaleStatus struct { + // actual number of observed instances of the scaled object. + Replicas int32 `json:"replicas"` + + // label query that should match the replicas count. This is same + // as the label selector but in the string format to avoid introspection + // by clients. The string will be in the same format as the query-param syntax. + // More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors + // +optional + Selector string `json:"selector"` +} +``` + +### Feature Gate + +The `SubResources` field in `CustomResourceDefinitionSpec` will be gated under the `CustomResourceSubResources` alpha feature gate. +If the gate is not open, the value of the new field within `CustomResourceDefinitionSpec` is dropped on creation and updates of CRDs. + +## Semantics + +### Validation Behavior + +#### Status + +The status endpoint of a CustomResource receives a full CR object. Changes outside of the status path are ignored. +For validation everything outside of the JSON Path `StatusPath` can be reset to the existing values of the CustomResource. Then the JSON Schema presented in the CRD is validated against the whole object. + +Note: one could extract the status subobject, filter the JSON schema by rules applied to the status and then validate the status subobject only. +The filter step of the JSON schema though is not obvious in the case that there are status validations not below a properties JSON Schema construct. For that reason, the alternative implementation is preferred as its semantics are much simpler. + +#### Scale + +Moreover, if the scale subresource is enabled: + +- The value at the specified JSON Path `SpecReplicasPath` (e.g. `.spec.replicas`) is validated to ensure that it contains non-negative integer value and is not empty. + +- The value at the optional JSON Path `StatusReplicasPath` (e.g. `.status.replicas`) is validated to be a an integer number if it exists (i.e. this can be empty). + +- The value at the optional JSON Path `LabelSelectorPath` (e.g. `.spec.labelSelector`) is validated to be a valid label selector if it exists (i.e. this can be empty). +If it is invalid, an empty `LabelSelector` is used. + +### Status Behavior + +If the `/status` subresource is enabled, the following behaviors change: + +- The main resource endpoint will ignore all changes in the specified status subpath. +(note: it will **not** reject requests which try to change the status, following the existing semantics of other resources). + +- The `.metadata.generation` field is updated if and only if the value at the specified `SpecPath` (e.g. `.spec`) changes. +Additionally, if the spec does not change, `.metadata.generation` is not updated. + +- The `/status` subresource receives a full resource object, but only considers the value at the specified `StatusPath` subpath (e.g. `.status`) for the update. +The value at the `.metadata` subpath is **not** considered for update as decided in https://github.com/kubernetes/kubernetes/issues/45539. + +Both the status and the spec (and everything else if there is anything) of the object share the same key in the storage layer, i.e. the value at `.metadata.resourceVersion` is increased for any kind of change. There is no split of status and spec in the storage layer. + +### Scale Behavior + +The number of CustomResources can be easily scaled up or down depending on the replicas field present in path specified by `SpecPath`. + +Only `ScaleSpec.Replicas` can be written. All other values are read-only and changes will be ignored. i.e. upon updating the scale subresource, two fields are modified: + +1. The replicas field is copied back from the `Scale` object to the main resource as specified by `SpecReplicasPath` in the CRD, e.g. `.spec.replicas = scale.Spec.Replicas`. + +2. The resource version is copied back from the `Scale` object to the main resource before writing to the storage: `.metadata.resourceVersion = scale.ResourceVersion`. +In other words, the scale and the CustomResource share the resource version used for optimistic concurrency. +Updates with outdated resource versions are rejected with a conflict error, read requests will return the resource version of the CustomResource. + +#### Status Replicas Behavior + +As only the `scale.Spec.Replicas` field is to be written to by the CR user, the user-provided controller (not any generic CRD controller) counts its children and then updates the controlled object by writing to the `/status` subresource, i.e. the `scale.Status.Replicas` field is read-only. + +#### Selector Behavior + +`CustomResourceSubResourceScale.LabelSelectorPath` is the label selector over CustomResources that should match the replicas count. +The value in the `Scale` object is one-to-one the value from the CustomResource if the label selector is non-empty. +Intentionally we do not default it to another value from the CustomResource (e.g. `.spec.template.metadata.labels`) as this turned out to cause trouble (e.g. in `kubectl apply`) and it is generally seen as a wrong approach with existing resources. + +## Implementation Plan + +The `/scale` and `/status` subresources are mostly distinct. It is proposed to do the implementation in two phases (the order does not matter much): + +1. `/status` subresource +2. `/scale` subresource + +## Alternatives + +### Scope + +In this proposal we opted for an opinionated concept of subresources i.e. we restrict the subresource spec to the two very specific subresources: `/status` and `/scale`. +We do not aim for a more generic subresource concept. In Kubernetes there are a number of other subresources like `/log`, `/exec`, `/bind`. But their semantics is much more special than `/status` and `/scale`. +Hence, we decided to leave those other subresources to the domain of User provided API Server (UAS) instead of inventing a more complex subresource concept for CustomResourceDefinitions. + +**Note**: that the types do not make the addition of other subresources impossible in the future. + +We also restrict the JSON path for the status and the spec within the CustomResource. +We could make them definable by the user and the proposed types actually allow us to open this up in the future. +For the time being we decided to be opinionated as all status and spec subobjects in existing types live under `.status` and `.spec`. Keeping this pattern imposes consistency on user provided CustomResources as well. \ No newline at end of file