Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring back iptable scanning #7723

Merged
merged 4 commits into from
Nov 12, 2024
Merged

Bring back iptable scanning #7723

merged 4 commits into from
Nov 12, 2024

Conversation

Nino-K
Copy link
Member

@Nino-K Nino-K commented Nov 5, 2024

Previously, the guestAgent in Rancher Desktop was picking up port mappings from iptables using the GetPorts function from Lima, as seen here. However, since the iptables scanning package was removed from the guestAgent, it no longer retrieves these port mappings.

An example of such a rule in iptables is as follows:

CNI-HOSTPORT-SETMARK  tcp  --  10.42.0.0/24         anywhere             tcp dpt:12345
CNI-HOSTPORT-SETMARK  tcp  --  localhost            anywhere             tcp dpt:12345
DNAT       tcp  --  anywhere             anywhere             tcp dpt:12345 to:10.42.0.100:80
CNI-DN-6ef6bede9244615191a81  tcp  --  anywhere             anywhere             /* dnat name: "cbr0" id: "0344293f74a8790f54a73bdd10f97bfaf98b3fb80b8e88c610f2da8ddf1c241a" */ multiport dports 12345

These rules are generated by the CNI (Container Network Interface) plugin for pod-to-pod communication. They are part of Kubernetes' networking and ensure that traffic on specific host ports (e.g., port 12345) is correctly routed to the appropriate pod, with the CNI plugin managing the handling of that traffic.

Given their role in routing traffic correctly within Kubernetes, it is important that these rules are not overlooked by the guestAgent.

TCP Repro Steps

  1. Download echo-server.yaml from https://gist.github.com/mikeseese/cca62f2dba7a453ebe172031a9490760
  2. Using Rancher Desktop v1.16.0, K8s v1.30.5, CE: moby, start up the cluster
  3. kubectl apply -f echo-server.yaml
  4. curl <hostIP>:12345/param?query=demo
  5. See that curl can't reach the host
    • It should have returned a JSON object, i.e.:
    {"host":{"hostname":"127.0.0.1","ip":"::ffff:127.0.0.1","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/param?query=demo","protocol":"http"},"request":{"params":{"0":"/param"},"query":{"query":"demo"},"cookies":{},"body":{},"headers":{"host":"127.0.0.1:12345","user-agent":"curl/8.1.2","accept":"*/*"}},"environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME":"echo-server","INGRESS_NGINX_CONTROLLER_ADMISSION_SERVICE_PORT_HTTPS_WEBHOOK":"443","INGRESS_NGINX_CONTROLLER_ADMISSION_PORT_443_TCP_ADDR":"10.43.14.22","INGRESS_NGINX_CONTROLLER_SERVICE_PORT_HTTP":"80","INGRESS_NGINX_CONTROLLER_PORT_443_TCP_PORT":"443","KUBERNETES_PORT_443_TCP_ADDR":"10.43.0.1","INGRESS_NGINX_CONTROLLER_ADMISSION_SERVICE_PORT":"443","INGRESS_NGINX_CONTROLLER_PORT_443_TCP":"tcp://10.43.38.178:443","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_SERVICE_PORT_HTTPS":"443","INGRESS_NGINX_CONTROLLER_SERVICE_HOST":"10.43.38.178","INGRESS_NGINX_CONTROLLER_SERVICE_PORT":"80","INGRESS_NGINX_CONTROLLER_PORT_80_TCP_ADDR":"10.43.38.178","INGRESS_NGINX_CONTROLLER_PORT_443_TCP_PROTO":"tcp","INGRESS_NGINX_CONTROLLER_ADMISSION_PORT_443_TCP_PORT":"443","INGRESS_NGINX_CONTROLLER_PORT_80_TCP":"tcp://10.43.38.178:80","INGRESS_NGINX_CONTROLLER_PORT_80_TCP_PROTO":"tcp","KUBERNETES_SERVICE_HOST":"10.43.0.1","KUBERNETES_PORT_443_TCP":"tcp://10.43.0.1:443","KUBERNETES_PORT_443_TCP_PORT":"443","INGRESS_NGINX_CONTROLLER_ADMISSION_SERVICE_HOST":"10.43.14.22","INGRESS_NGINX_CONTROLLER_ADMISSION_PORT":"tcp://10.43.14.22:443","INGRESS_NGINX_CONTROLLER_ADMISSION_PORT_443_TCP":"tcp://10.43.14.22:443","INGRESS_NGINX_CONTROLLER_SERVICE_PORT_HTTPS":"443","INGRESS_NGINX_CONTROLLER_PORT_80_TCP_PORT":"80","KUBERNETES_PORT_443_TCP_PROTO":"tcp","INGRESS_NGINX_CONTROLLER_ADMISSION_PORT_443_TCP_PROTO":"tcp","INGRESS_NGINX_CONTROLLER_PORT":"tcp://10.43.38.178:80","INGRESS_NGINX_CONTROLLER_PORT_443_TCP_ADDR":"10.43.38.178","KUBERNETES_PORT":"tcp://10.43.0.1:443","NODE_VERSION":"16.16.0","YARN_VERSION":"1.22.19","HOME":"/root"}}
    
  6. To see what you should have seen, you can forward the port manually: kubectl port-forward pods/echo-server 12345:80 -n default and change the curl command to use the localhost IP curl 127.0.0.1:12345/param?query=demo

Fixes: #7722

@Nino-K Nino-K force-pushed the bring-back-iptable-scanning branch from 7b09ab6 to ab27674 Compare November 5, 2024 20:34
@jandubois
Copy link
Member

Previously, the guestAgent in Lima was picking up port mappings from iptables using the GetPorts function, as seen here. However, since the iptables scanning package was removed from the guestAgent, it no longer retrieves these port mappings.

The iptables scanning code was never removed from Lima; it is still there. Did you mean the code in Rancher Desktop was removed from the WSL2 code?

Copy link
Member

@jandubois jandubois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly comments, but I do think the blocking sleep must be replaced by something cancellable.

portMap := make(nat.PortMap)
// Add new forwards
for _, p := range added {
if p.TCP {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only TCP? I thought we also do UDP, or am I confused?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We totally support UDP now in RD but, this decision came from how the iptables.Entry defined in the upstream. It doesn't seem to give the proper protocol but rather a boolean. Of course, if TCP is set to true, we know it is 100% TCP but when it's set to false I'm not sure if that means it would be UDP 100%.

@Nino-K
Copy link
Member Author

Nino-K commented Nov 7, 2024

The iptables scanning code was never removed from Lima; it is still there. Did you mean the code in Rancher Desktop was removed from the WSL2 code?

Thanks, I corrected it now. I meant Rancher Desktop.

@Nino-K
Copy link
Member Author

Nino-K commented Nov 7, 2024

Mostly comments, but I do think the blocking sleep must be replaced by something cancellable.

I agree. To be honest, I just revived the old code with a minor modification to make it work for our current needs, but I didn't change any of the old pars, assuming that it was previously doing the right thing. However, I agree with removing the time.Sleep. Also, the comparePorts code naming is terrible.

Previously, the guestAgent in Rancher Desktop was picking up port mappings from iptables using the GetPorts function, as seen [here](https://github.com/lima-vm/lima/blob/3e01f2597cba83914e0dcd8ff3b87a409ca8e0ae/pkg/guestagent/iptables/iptables.go#L36). However, since the iptables scanning package was removed in commit [ca0009c](ca0009c) from the guestAgent, it no longer retrieves these port mappings.

An example of such a rule in iptables is as follows:

```
CNI-HOSTPORT-SETMARK  tcp  --  10.42.0.0/24         anywhere             tcp dpt:12345
CNI-HOSTPORT-SETMARK  tcp  --  localhost            anywhere             tcp dpt:12345
DNAT       tcp  --  anywhere             anywhere             tcp dpt:12345 to:10.42.0.100:80
CNI-DN-6ef6bede9244615191a81  tcp  --  anywhere             anywhere             /* dnat name: "cbr0" id: "0344293f74a8790f54a73bdd10f97bfaf98b3fb80b8e88c610f2da8ddf1c241a" */ multiport dports 12345
```
The rules above are custom rules generated by the CNI plugin to manage traffic to the cbr0 network bridge, the default bridge network interface Kubernetes uses for pod-to-pod communication. These rules seem to be a part of Kubernetes’ networking setup.

Given that these rules are essential for Kubernetes networking—ensuring traffic on specific hostports (e.g., port 12345) is correctly routed to a pod and handled by the CNI plugin—it seems important that they are not ignored.

Signed-off-by: Nino Kodabande <[email protected]>
Signed-off-by: Nino Kodabande <[email protected]>
@Nino-K Nino-K force-pushed the bring-back-iptable-scanning branch from ab27674 to da1ac1c Compare November 7, 2024 18:50
@Nino-K Nino-K requested a review from jandubois November 7, 2024 18:52
Copy link
Member

@jandubois jandubois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

Just one more stylistic tweak please!

Comment on lines 45 to 47
select {
case <-ticker.C:
// Detect ports for forward
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of having a 60-line case statement. Wouldn't it be better to have a minimal select, and then have the rest of the logic as the body of the for loop:

Suggested change
select {
case <-ticker.C:
// Detect ports for forward
select {
case <-ctx.Done():
return nil
case <-ticker.C:
}
// Detect ports for forward

@Nino-K Nino-K requested a review from jandubois November 12, 2024 18:19
Copy link
Member

@jandubois jandubois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM

@jandubois jandubois merged commit eca49fb into main Nov 12, 2024
31 checks passed
@jandubois jandubois deleted the bring-back-iptable-scanning branch November 12, 2024 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GuestAgent is ignoring iptable portMappings in CNI-DN
2 participants