Initial AWS EKS support

[timberio-campfire]

We'd like to officially support Amazon's EKS platform.

1. We should start with a specification to determine the best path forward.

2. Its possible this might just wrap existing sources.

3. Its possible this might just be an installation page in our documentation.

[timberio-campfire]

Link to feature: https://timber.productboard.com/feature-board/planning/features/2784927

[LucioFranco]

@binarylogic I think this makes sense, do we also want to open a more generic kube issue since I believe the implementation that will work for eks will work for any kube deployment?

[binarylogic]

Yes, #260 handles that.

[ktff]

Current Kubernetes implementation in #893 was tested with AWS EKS ec2 cluster, and it works out of the box. Except of one bug found during testing.

So as @binarylogic said, this will be solved with #260. I anticipate that only retesting it on EKS will be needed.

[LucioFranco]

There is nothing to spec out here. Our current kube implementation covers this. What we will need is deployment guides around kube in general and these can be applied to EKS just like how you might apply it to any other kube cluster. Based on that I think this issue can be closed technically?

cc @binarylogic

[ktff]

I wouldn't close this issue until #1060 is implemented. The reason being that namespace information is necessary on EKS to properly filter out system logs. #893 on it's own isn't capable of doing that on EKS. Although, the number of system logs isn't large, it would still bother users.

[LucioFranco]

@ktff I've updated #1060 to include that additional filter.

[alexgavrisco]

Is there anything Vector is missing for AWS EKS and it should be considering when deploying there? I've deployed it to AWS EKS (on managed worker nodes) and didn't notice any issues so far. However, it's still being validated in a staging environment.

[ktff]

@Alexx-G No, everything is in place, not only for EKS, but basically for any Kubernetes, with exception of Kubernetes backed by Windows nodes which they started recently supporting. But I would wait for 0.9 release as there is a breaking change in how Vector should be run, more specifically #2188.

Also this #1867 will change how the fields are named.

[alexgavrisco]

@ktff Thanks for details! I was expecting 0.9 to have breaking changes and there's a number of fixes I need, so this is fine.
Since #1888 has been merged, I suppose after 0.9 you don't expect any breaking changes (at least for event schema for kubernetes source)?

[binarylogic]

@Alexx-G that's the goal. 0.10.0 is the "Kubernetes Release", where we will officially announce the integration. I don't foresee any major changes between 0.9.0 and 0.10.0, but we will also be rigorously QA'ing the integration during that time.

[MOZGIII]

We should verify things work as expected before we consider this done.

[binarylogic]

Just clarifying the work here. We should verify and test this, possibly even add an integration test, but we should defer for it now. We have an epic to verify integration with a variety of hosted K8s platforms.

[MOZGIII]

I've started working is this. Apparently EKS integrations breaks in unexpected places (i.e. elasticsearch source), so I'll create issues to fix what I've found.

[Dragotic]

Hello, any news about the AWS EKS Support?

[jszwedko]

Hi @Dragotic !

I think this may actually already be done since we do support deploying Vector to EKS. We have published Helm charts that make this easier. Is there anything in particular that you find missing though? We do have people successfully running Vector on EKS.

[jszwedko]

I'll close this since we do have people running Vector on EKS, but we can open up follow-up issues if there is any support lacking.