diff --git a/keps/sig-windows/20190103-windows-node-support.md b/keps/sig-windows/20190103-windows-node-support.md index 47b6e0241e0..02f9476a644 100644 --- a/keps/sig-windows/20190103-windows-node-support.md +++ b/keps/sig-windows/20190103-windows-node-support.md @@ -4,6 +4,7 @@ authors: - "@astrieanna" - "@benmoss" - "@patricklang" + - "@michmike" owning-sig: sig-windows participating-sigs: - sig-architecture @@ -11,11 +12,16 @@ participating-sigs: reviewers: - sig-architecture - sig-node + - sig-testing + - sig-release approvers: - "@bgrant0607" + - "@michmike" + - "@patricklang" + - "@spiffxp" editor: TBD creation-date: 2018-11-29 -last-updated: 2019-01-21 +last-updated: 2019-01-25 status: provisional --- @@ -55,17 +61,18 @@ There is strong interest in the community for adding support for workloads runni ## Motivation -Windows-native workloads still account for a significant portion of the enterprise software space. While containerization technologies emerged first in the UNIX ecosystem, Microsoft has made investments in recent years to enable support for containers in its Windows OS. As users of Windows increasingly turn to containers as the preferred abstraction for running software, the Kubernetes ecosystem stands to benefit by becoming a cross-platform cluster manager. +Windows-based workloads still account for a significant portion of the enterprise software space. While containerization technologies emerged first in the UNIX ecosystem, Microsoft has made investments in recent years to enable support for containers in its Windows OS. As users of Windows increasingly turn to containers as the preferred abstraction for running software and modernizing existing applications, the Kubernetes ecosystem stands to benefit by becoming a cross-platform cluster manager. ### Goals -- Enable users to run nodes on Windows servers +- Enable users to schedule Windows Server containers in Kubernetes through the introduction of support for Windows compute nodes - Document the differences and limitations compared to Linux -- Test results added to testgrid to prevent regression of functionality +- Create a test suite in testgrid to maintain high quality for this feature and prevent regression of functionality ### Non-Goals - Adding Windows support to all projects in the Kubernetes ecosystem (Cluster Lifecycle, etc) +- Enable the Kubernetes master components to run on Windows ## Proposal @@ -73,21 +80,33 @@ As of 29-11-2018 much of the work for enabling Windows nodes has already been co ### What works today - Windows-based containers can be created by kubelet, [provided the host OS version matches the container base image](https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility) - - ConfigMap, Secrets: as environment variables or volumes + - Pod (single or multiple containers per Pod with process isolation), Deployment, ReplicaSet + - Services types NodePort, ClusterIP, LoadBalancer, and ExternalName + - ConfigMap, Secrets: as environment variables or volumes - Resource limits - Pod & container metrics -- Pod networking with [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), [OVN-Kubernetes](https://github.com/openvswitch/ovn-kubernetes), [two CNI meta-plugins](https://github.com/containernetworking/plugins), [Flannel](https://github.com/coreos/flannel) and [Calico](https://github.com/projectcalico/calico) + - Horizontal Pod Autoscaling +- Windows Server 2019 is the only Windows operating system we will support at GA timeframe. Note above that the host operating system version and the container base image need to match. This is a Windows limitation we cannot overcome. +- Customers can deploy a heterogeneous cluster, with Windows and Linux compute nodes side-by-side and schedule Docker containers on both operating systems. Of course, Windows Server containers have to be scheduled on Windows and Linux containers on Linux +- Out-of-tree Pod networking with [Azure-CNI](https://github.com/Azure/azure-container-networking/blob/master/docs/cni.md), [OVN-Kubernetes](https://github.com/openvswitch/ovn-kubernetes), [two CNI meta-plugins](https://github.com/containernetworking/plugins), [Flannel (VXLAN and Host-Gateway)](https://github.com/coreos/flannel) - Dockershim CRI - Many[1] of the e2e conformance tests when run with [alternate Windows-based images](https://hub.docker.com/r/e2eteam/) which are being moved to [kubernetes-sigs/windows-testing](https://www.github.com/kubernetes-sigs/windows-testing) - Persistent storage: FlexVolume with [SMB + iSCSI](https://github.com/Microsoft/K8s-Storage-Plugins/tree/master/flexvolume/windows), and in-tree AzureFile and AzureDisk providers +- Windows Server containers can take advantage of StatefulSet functionality for stateful applications and distributed systems +- Windows Pods can take advantage of DaemonSet, with the exception that privileged containers are not supported on Windows (more on that below) ### What will work eventually +- Group Managed Service Accounts, a way to assign an Active Directory identity to a Windows container, is forthcoming with KEP `Windows Group Managed Service Accounts for Container Identity` - `kubectl port-forward` hasn't been implemented due to lack of an `nsenter` equivalent to run a process inside a network namespace. - CRIs other than Dockershim: CRI-containerd support is forthcoming +- Some kubeadm work was done in the past to add Windows nodes to Kubernetes, but that effort has been dormant since. We will need to revisit that work and complete it in the future. +- Calico CNI for Pod networking +- Hyper-V isolation (Currently this is limited to 1 container per Pod and is an alpha feature) +- It is unclear if the RuntimeClass proposal from sig-node will simplify scheduled Windows containers. we will work with sig-node on this. ### What will never work (without underlying OS changes) - Certain Pod functionality - - Privileged containers + - Privileged containers and other Pod security context privilege and access control settings - Reservations are not enforced by the OS, but overprovisioning could be blocked with `--enforce-node-allocatable=pods` (pending: tests needed) - Certain volume mappings - Single file & subpath volume mounting @@ -96,8 +115,9 @@ As of 29-11-2018 much of the work for enabling Windows nodes has already been co - readOnly root filesystem. Mapped volumes still support readOnly - Termination Message - these require single file mappings - CSI plugins, which require privileged containers +- Host networking is not available in Windows - [Some parts of the V1 API](https://github.com/kubernetes/kubernetes/issues/70604) -- Overlay networking support in Windows Server 1803 is not fully functional using the `win-overlay` CNI plugin. Specifically service IPs do not work on Windows nodes. This is currently specific to `win-overlay` - other CNI plugins (OVS, AzureCNI) work. +- Overlay networking support in Windows Server 1803 is not fully functional using the `win-overlay` CNI plugin. Specifically service IPs do not work on Windows nodes. This is currently specific to `win-overlay`; other CNI plugins (OVS, AzureCNI) work. Since Windows Server 1803 is not supported for GA, this is mostly not applicable. We left it here since it impacts beta ### Relevant resources/conversations @@ -110,13 +130,51 @@ As of 29-11-2018 much of the work for enabling Windows nodes has already been co **Second class support**: Kubernetes contributors are likely to be thinking of Linux-based solutions to problems, as Linux remains the primary OS supported. Keeping Windows support working will be an ongoing burden potentially limiting the pace of development. -**User experience**: Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads separated. In the best case this imposes a burden only on Windows users, but this is still less than ideal. +**User experience**: Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads separated. In the best case this imposes a burden only on Windows users, but this is still less than ideal. The recommended approach is outlined below -## Graduation Criteria +#### Ensuring OS-specific workloads land on appropriate container host +As you can see below, we plan to document how Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All nodes today have the following default labels +- beta.kubernetes.io/os = [windows|linux] +- beta.kubernetes.io/arch = [amd64|arm64|...] +If a deployment does not specify a nodeSelector like `"beta.kubernetes.io/os": windows`, it is possible the Pods can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice we will recommend is to use a nodeSelector. -## Implementation History +However, we understand that in certain cases customers have a pre-existing large number of deployments for Linux containers. Since they will not want to change all deployments to add nodeSelectors, the alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only (`“--register-with-taints=’os=Win1809:NoSchedule’” `). By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector to choose Windows, and a toleration. +``` +nodeSelector: + "beta.kubernetes.io/os": windows +tolerations: + - key: "os" + operator: "Equal" + Value: “Win1809” + effect: "NoSchedule" +``` +## Graduation Criteria +- All features and functionality under `What works today` is fully tested and vetted to be working by SIG-Windows +- SIG-Windows has high confidence to the stability and reliability of Windows Server containers on Kubernetes +- 100% green/passing conformance tests that are applicable to Windows (see the Testing Plan section for details on these tests) +- Comprehensive documentation that includes but is not limited to the following sections. Documentation will reside at https://kubernetes.io/docs +1. Outline of Windows Server containers on Kubernetes +2. Getting Started Guide, including Prerequisites +3. How to deploy Windows nodes in Kubernetes +4. Overview of Networking on Windows +5. Links to documentation on how to deploy and use CNI plugins for Windows (example for OVN - https://github.com/openvswitch/ovn-kubernetes/tree/master/contrib) +6. Links to documentation on how to deploy Windows nodes for public cloud providers or other Kubernetes distributions (example for Rancher - https://rancher.com/docs//rancher/v2.x/en/cluster-provisioning/rke-clusters/windows-clusters/) +7. How to schedule Windows Server containers, including examples +8. Advanced: How to use metrics and the Horizontal Pod Autoscaler +9. Advanced: How to use Group Managed Service Accounts +10. Advanced: How to use Taints and Tolerations for a heterogeneous compute cluster (Windows + Linux) +11. Advanced: How to use Hyper-V isolation (not a stable feature yet) +12. Advanced: How to build Kubernetes for Windows from source +13. Supported functionality (with examples where appropriate) +14. Known Limitations +15. Unsupported functionality +16. Resources for contributing and getting help - Includes troubleshooting help and links to additional troubleshooting guides like https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/common-problems + +## Implementation History +- Alpha was released with Kubernetes v.1.5 +- Beta was released with Kubernetes v.1.9 ## Testing Plan