Proposal for growing FlexVolume size.

Proposal for growing FlexVolume size, the feature ticket is at:
kubernetes/enhancements#304

The original google doc for this proposal is at:
https://docs.google.com/document/d/1dwom9xQ3Fg5F_jJrybr0slp-QsO3_CKiisxzNRoMhec/edit?usp=sharing

contributors/design-proposals/storage/grow-flexvolume-size.md

@@ -0,0 +1,200 @@
+# Proposal for Growing FlexVolume Size
+
+**Authors:** [xingzhou](https://github.com/xingzhou)
+
+## Goals
+
+Since PVC resizing is introduced in Kubernetes v1.8, several volume plugins have already supported this feature, e.g. GlusterFS, AWS EBS. In this proposal, we are proposing to support FlexVolume expansion. So when user uses FlexVolume and corresponding volume driver to connect to his/her backend storage system, he/she can expand the PV size by updating PVC in Kubernetes.
+
+## Non Goals
+
+* We only consider expanding FlexVolume size in this proposal. Decreasing size of FlexVolume will be designed in the future.
+* In this proposal, user can only expand the FlexVolume size manually by updating PVC. Auto-expansion of FlexVolume based on specific meterings is not considered.
+* The proposal only contains the changes made in FlexVolume, volume driver changes which should be made by user are not included.
+
+## Implementation Designs
+
+### Prerequisites
+
+* Kubernetes should be at least v1.8.
+* Enable resizing by setting feature gate `ExpandPersistentVolumeGate` to `true`.
+* Enable `PersistentVolumeClaimResize` admission plugin(optional).
+* Follow the UI of PV resizing, including:
+  * Only dynamic provisioning supports volume resizing
+  * Set StorageClass attribute `allowVolumeExpansion` to `true`
+
+### Admission Control Changes
+
+Whether or not a specific volume plugin supports volume expansion is validated and checked in PV resize admission plugin. In general, we can list FlexVolume as the ones that support volume expansion and leave the actual expansion capability check to the underneath volume driver when PV resize controller calls the `ExpandVolumeDevice` method of FlexVolume.
+
+In PV resize admission plugin, add the following check to `checkVolumePlugin` method:
+```
+// checkVolumePlugin checks whether the volume plugin supports resize
+func (pvcr *persistentVolumeClaimResize) checkVolumePlugin(pv *api.PersistentVolume) bool {
+  ...
+  if pv.Spec.FlexVolume != nil {
+    return true
+  }
+  ...
+}
+```
+
+### FlexVolume Plugin Changes
+
+FlexVolume relies on underneath volume driver to implement various volume functions, e.g. attach/detach. As a result, volume driver will decide whether volume can be expanded or not. By calling the `init` method of volume driver, the FlexVolume can detect the volume driver’s capabilities. To support volume expansion, we need to add two new capabilities to FlexVolume: 
+* `Expand`: identifies whether or not the underneath volume driver supports volume expansion.
+* `RequiresFSResize`: if volume driver supports expansion, identifies whether or not the underneath volume driver requires file system resize after the physical volume is expanded. Note that if this value is set to `true`, resize controller will leave FS resize to the kubelet on worker node and kubelet will call FlexVolume `ExpandFS` method to finish the file system resize.
+
+FlexVolume type `DriverCapability` can be redefined as:
+```
+type DriverCapabilities struct {
+  Attach           bool `json:"attach"`
+  Expand           bool `json:"expand"`
+  RequiresFSResize bool `json:"requiresFSResize"`
+  SELinuxRelabel   bool `json:"selinuxRelabel"`
+}
+ 
+func defaultCapabilities() *DriverCapabilities {
+  return &DriverCapabilities{
+    Attach:           true,
+    Expand:           false,//By default, we disable the expand capability
+    RequiresFSResize: true, //By default, we require file system resize which will be done by kubelet 
+    SELinuxRelabel:   true,
+  }
+}
+```
+
+At present, a FlexVolume plugin can have two types, one is normal volume plugin while the other one is attachable volume plugin. These two types of plugin are defined as:
+```
+// FlexVolumePlugin object.
+type flexVolumePlugin struct {
+  driverName string
+  execPath   string
+  host       volume.VolumeHost
+  runner     exec.Interface
+ 
+  sync.Mutex
+  unsupportedCommands []string
+  capabilities        DriverCapabilities
+}
+ 
+type flexVolumeAttachablePlugin struct {
+  *flexVolumePlugin
+}
+```
+
+To support volume expansion, FlexVolume need to define two new types of volume plugin, one is expandable volume plugin and the other one is expandable & attachable volume plugin (which can support the volume driver with both volume attach/detach and expansion capabilities), these two new types are like:
+```
+type flexVolumeExpandablePlugin struct {
+  *flexVolumePlugin
+}
+ 
+type flexVolumeAttachableAndExpandablePlugin struct {
+  *flexVolumePlugin
+  *flexVolumeAttachablePlugin
+  *flexVolumeExpandablePlugin
+}
+```
+
+FlexVolume shall create and return different types of volume plugin objects according to underneath volume driver’s capabilities (detected by volume driver `init` method)
+
+The new `flexVolumeExpandablePlugin` type must implement the following `ExpandableVolumePlugin` interfaces:
+
+#### ExpandVolumeDevice
+
+Volume resizing controller invokes this method while receiving a valid PVC resizing request. FlexVolume plugin calls the underneath volume driver’s corresponding `expandvolume` method with three parameters, including new size of volume(number in bytes), old size of volume(number in bytes) and volume spec, to expand PV. Once the expansion is done, volume driver should return the new size(number in bytes) of the volume to FlexVolume.
+
+A sample implementation of `ExpandVolumeDevice` method is like:
+```
+func (plugin *flexVolumeExpandablePlugin) ExpandVolumeDevice(spec *volume.Spec, newSize resource.Quantity, oldSize resource.Quantity) (resource.Quantity, error) {
+  const timeout = 10*time.Minute
+ 
+  call := plugin.NewDriverCallWithTimeout(expandVolumeCmd, timeout)
+  call.Append(newSize.Value())
+  call.Append(oldSize.Value())
+  call.AppendSpec(spec, plugin.host, nil)
+ 
+  ds, err := call.Run()
+  if err != nil {
+      return resource.NewQuantity(0, resource.BinarySI), err
+  }
+ 
+  return  resource.NewQuantity(ds.ActualVolumeSize, resource.BinarySI), nil
+}
+```
+
+Add a new field in type `DriverStatus` named `ActualVolumeSize` to identify the new expanded size of the volume returned by underneath volume driver:
+```
+// DriverStatus represents the return value of the driver callout.
+type DriverStatus struct {
+  ...
+  ActualVolumeSize int64 `json:"volumeNewSize,omitempty"`
+}
+```
+
+#### RequiresFSResize
+
+`RequiresFSResize` is a method to implement `ExpandableVolumePlugin` interface. The return value of this method identifies whether or not a file system resize is required once physical volume get expanded. If the return value is `true`, PV resize controller will consider the volume resize operation is done and then update the PV object’s capacity in K8s directly; If the return value is `false`, PV resize controller will leave kubelet to do the file system resize, and kubelet on worker node will call `ExpandFS` method of FlexVolume to finish the file system resize step(at present, only offline FS resize is supportted, online resize support is under community discussion [here](https://github.com/kubernetes/community/pull/1535)). 
+
+The return value of `RequiresFSResize` is collected from underneath volume driver when FlexVolume invokes `init` method of volume driver. The sample code of `RequiresFSResize` in FlexVolume looks like:
+```
+func (plugin *flexVolumeExpandablePlugin) RequiresFSResize() bool {
+  return plugin.capabilities.RequiresFSResize
+}
+```
+
+#### ExpandFS
+
+`ExpandFS` is another method to implement `ExpandableVolumePlugin` interface. This method allows volume plugin itself instead of kubelet to resize the file system. If volume plugin returns `true` for `RequiresFSResize`, PV resize controller will leave FS resize to kubelet on worker node. Kubelet then will call FlexVolume `ExpandFS` to resize file system once physical volume expansion is done.
+
+As `ExpandFS` is called on worker node, volume driver can also take this chance to do physical volume resize together with file system resize as well. Also, current code only supports offline FS resize, online resize support is under dicsussion [here](https://github.com/kubernetes/community/pull/1535). Once online resize is implemented, we can also leverage online resize for FlexVolume by `ExpandFS` method.
+
+Note that `ExpandFS` is a new API for `ExpandableVolumeDriver`, the community ticket can be found [here](https://github.com/kubernetes/kubernetes/issues/58786).
+
+`ExpandFS` will call underneath volume driver `expandfs` method to finish FS resize. The sample code looks like:
+```
+func (plugin *flexVolumeExpandablePlugin) ExpandFS(spec *volume.Spec, newSize resource.Quantity, oldSize resource.Quantity)  error {
+  const timeout = 10*time.Minute
+ 
+  call := plugin.NewDriverCallWithTimeout(expandFSCmd, timeout)
+  call.Append(newSize.Value())
+  call.Append(oldSize.Value())
+  call.AppendSpec(spec, plugin.host, nil)
+ 
+  _, err := call.Run()
+ 
+  return  err
+}
+```
+
+For more design and details on how kubelet resizes volume file system, please refer to volume resizing proposal at:
+https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/grow-volume-size.md
+
+### Volume Driver Changes
+
+The volume driver that supports volume expansion shall report the `expand` capability in its `init` method. 
+
+Here is a shell script sample of the volume driver that supports expansion:
+```
+if [ "$op" = "init" ]; then
+        log '{"status": "Success", "capabilities": {"expand": true, “requiresFSResize”: true}}'
+        exit 0
+fi
+```
+
+In addition, the volume driver also needs to implement two new interfaces: `expandvolume` and `expandfs`
+
+For `expandvolume`, it takes three parameters: new size of volume(number in bytes), old size of volume(number in bytes) and volume spec json string. `expandvolume` expands the physical backend volume size and return the new size(number in bytes) of volume.
+
+For those volume plugins who need file system resize after physical volume is expanded, the `expandfs` method can take the FS resize work. If volume driver returns `true` for `RequiresFSResize`, this method will be called from kubelet on worker node. Volume driver can do the file system resize (or physical volume resize together with file system resize) inside this method
+
+### UI
+
+Expand FlexVolume size follows the same process as expanding other volume plugins, like GlusterFS. User creates and binds PVC and PV first. Then by using `kubectl edit pvc xxx` command, user can update the new size of PVC.
+
+## References
+
+* [Proposal for Growing Persistent Volume Size](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/grow-volume-size.md)
+* [PR for Volume Resizing Controller](https://github.com/kubernetes/kubernetes/commit/cd2a68473a5a5966fa79f455415cb3269a3f7462)
+* [Online FS resize support](https://github.com/kubernetes/community/pull/1535)
+* [Add “ExpandFS” method to “ExpandableVolumePlugin” interface](https://github.com/kubernetes/kubernetes/issues/58786)