Windows support

[shirwahersi]

Tell us about your request
Windows OS support

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We have mixed OS workload (Linux,Windows), we would like to managed both Linux/Windows nodes with Karpenter

[ellistarn]

This should be straightforward to support as a new value for key kubernetes.io/os.

[github-actions]

This issue is stale because it has been open 25 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[shirwahersi]

Hi all, Is there any plans to support Windows in near future?

[ellistarn]

Some folks were looking at contributing it, but I'm not sure where they're at.

[halovanic]

I believe @bwagner closed this patch attempt for the understandable difficulties keeping up with other changes around AMI handling in Karpenter.
Having gone back over the patches to test on my EKS cluster with post-0.15, we could aim for very basic support that gets Windows working, just not as its own first-class AMI Family:

1. Support vpc.amazonaws.com/PrivateIPv4Address resources that are needed for Windows-scheduled pods
2. Remove or replace a hardcoded requirement getting added in the instance type calculation that forbids non-Linux OSs

This does require the user to define a launch template in the Windows provisioner, but this is the same requirement Windows EKS users already had with Self-Managed node groups. I also think a Windows provisioner probably should define the system reserved memory and restrict instance types more explicitly, but the effort is still quite minimal.

This has worked out quite well in my admittedly limited tests: without even really understanding anything specific about Windows, Karpenter seems to have done a more effective job of getting all of my large workloads scheduled.

Is there interest in pursuing these more limited changes? I can submit reworked patches if there is.

[ellistarn]

define a launch template in the Windows provisioner

We'll likely be deprecating custom launch template support. Any chance you use can amiSelector and userData to meet the use case instead?

Is there interest in pursuing these more limited changes?

Yes! Happy to expand non-linux OS. Can you elaborate on PrivateIPv4Address?

[halovanic]

That would require a new user data format and corresponding AMI Family defined. That was already in the original patches, but seemed to be the part that was running into difficulties with a fast moving Karpenter target state.
What I am proposing are to tackle these more stable prerequisites for Windows separately, then we could come back to dedicated Windows AMI support before the launch template support goes away,

PrivateIPv4Address support was in the previous patches (see function awsPodPrivateIPv4() in either) and is needed because Windows supports fewer pods and EKS's webhook adds this calculation to prevent overscheduling.

[bwagner5]

This may just work with Custom AMIs (not AMI Families) https://karpenter.sh/v0.16.0/aws/user-data/#custom-amis

You should be able to specify an arbitrary AMI and user-data which will be passed directly to EC2. You could also set the max-pods on the Provisioner spec to deal with the PrivateIPv4Address in the mean time at least. https://karpenter.sh/v0.16.0/provisioner/#:~:text=storage%3A%202Gi-,maxPods%3A%2020,-%23%20Resource%20limits%20constrain

[halovanic]

I will do some further tests to see if I can switch over to Custom AMIs from the launch templates in my end-to-end tests; I don't see anything that would block it.

Just to clarify, the need for PrivateIPv4Address is not directly one of pod density; it's that the AWS VPC Resource Controller forces this on scheduled pods. This was the only reference I could actually find on this behavior:
https://github.com/aws/amazon-vpc-resource-controller-k8s/blob/master/docs/windows/workflow.md

[marciogmorales]

Hello Karpenter experts, any update in this topic?

[pdavagia]

Checking in to see if there are any updates on this?

[DorukAkinci]

We have now managed Windows Node groups support on the EKS service. Does that mean this implementation/feature may also come soon? aws/containers-roadmap#584