Magicdns #79

aojea · 2024-10-27T20:39:03Z

No description provided.

justinsb · 2024-11-05T01:11:23Z

cmd/kindnetd/dnsproxy.go

+	}
+
+	// create a connection with the upstream DNS server
+	dnsTCPConn, err := dialer.Dial("tcp", net.JoinHostPort(d.nameServer, "53"))


It's great that this upstreams over TCP!

justinsb · 2024-11-05T01:14:00Z

cmd/kindnetd/dnsproxy.go

+	if len(questions) == 0 {
+		return dnsErrorMessage(hdr.ID, dnsmessage.RCodeFormatError, dnsmessage.Question{})
+	}
+	// it is supported but not wildly implemented, at least not in golang stdlib


Nit: s/wildly/widely

justinsb · 2024-11-05T12:33:32Z

cmd/kindnetd/dnsproxy.go

+	}
+	switch q.Type {
+	case dnsmessage.TypeA:
+		klog.V(7).Infof("DNS A request for %s", q.Name.String())


I'd be really interested to see what DNS requests come from your average pod. If we could "fix" the ndots situation, that would be great. That's got me thinking that we might want to (somehow?) warn about namespaces that collide with TLDs; I quickly scanned through B - both .blog and .build are TLDs and not-unlikely namespace-names!

this was reported recently in sig-network, I think you want something like https://coredns.io/plugins/kubernetes/#autopath , but once we have tcp pipelining working with a pool of connections I'm curious about if that will be relevant ... from 5 requests 4 will be nxdomains

justinsb · 2024-11-05T12:35:26Z

cmd/kindnetd/dnsproxy.go

+	switch q.Type {
+	case dnsmessage.TypeA:
+		klog.V(7).Infof("DNS A request for %s", q.Name.String())
+		addrs, err := d.lookupIP(context.Background(), "ip4", q.Name.String())


Idea for the future: I do think that sometimes DNS returns additional records, to avoid a round-trip. Maybe lookupIP should return a set of DNS answers.

yeah, I think what I'm doing have some limitations, and I just should work with the dns records directly instead of perform the translation

justinsb · 2024-11-05T12:55:36Z

cmd/kindnetd/dnsproxy.go

+		PreferGo: true,
+		Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
+			select {
+			case conn, ok := <-d.connCh:


Nit: I do wonder if we should put the dial logic in this function, so we can recover if we drop a TCP connection? I assume we don't because we want to reuse it?

This turns out to be more involved https://datatracker.ietf.org/doc/html/rfc7766#ref-edns-tcp-keepalive , we want to keep a pool of connections and we need to work on keep them open, then we can implement pipelining

justinsb · 2024-11-05T12:57:12Z

cmd/kindnetd/dnsproxy.go

+		LocalAddr: &net.UDPAddr{IP: net.ParseIP(d.nameServer), Port: 53},
+		Control: func(network, address string, c syscall.RawConn) error {
+			return c.Control(func(fd uintptr) {
+				// Mark connections so thet are not processed by the netfilter TPROXY rules


Nit: "... so that they are not processed ... " ?

justinsb · 2024-11-05T12:58:13Z

cmd/kindnetd/dnsproxy.go

+	klog.V(7).Info("starting parsing packet")
+	hdr, err := p.Start(b)
+	if err != nil {
+		return dnsErrorMessage(hdr.ID, dnsmessage.RCodeFormatError, dnsmessage.Question{})


Nit: maybe logging for the error cases here?

justinsb · 2024-11-05T13:01:12Z

aojea · 2024-12-24T08:15:18Z

@justinsb I have this fundamental question? should we cache or should we serve the DNS directly from each Nodes ... imaging we embed kube-proxy and coredns, both reuse the same information for EndpointSlices and Services and you save 2 components ... and kindnet becomes the only networking component needed in the cluster ... my point is that I don't see much value in caching if you can serve directly from the nodes ... the later completely removes the problem meanwhile the former just mitigates it but introduces a complexity that can be hard to support ... transparent proxies are complicated if they start misbehaving

aojea · 2025-01-10T23:40:28Z

This is next , I just settled on the algorithm, the main problem of kubernetes DNS is p99 latency and conntrack exhaustion, to solve that kindnet will

use nfqueue to divert the pods udp packets destined to port 53 to userspace (TCP does not have conntrack problems and will not benefit of caching)
use a TCP persistent connection with the upstream DNS server (dnsmasq) has problem with the number of TCP connections so we'll add metrics to the latency to evaluate the number of connections needed with the upstream server
-it will only process A and AAAA records to begin with, we are going to add metrics to get stats of all types of records to reevaluate later (SVR, PTR and CNAME may be interesting), if is not an A or an AAAA record it will just pass the packet to the kernel again
it will check the cache (will keep 30s by record same as GKE is using and a fixed size of cache using LRU with 1024 records to avoid memory consumption) and reply if is present
if is not present in the cache, it will use the upstream TCP connection to query the apiserver

enable dns cache by default, it has the proper guardrails to avoid to break clusters, if anything fails processing the packet it falls back to the dataplane.

aojea · 2025-01-12T21:03:36Z

@justinsb Quick test with https://github.com/guessi/dnsperf-bench
I'm super happy how this works now, it has good guardrails #79 (comment), it allows to cache requests and to reduce the conntrack entries, also provide great visibility on the protocol with good metrics, number of requests and latency per request and type of record requested.

DNS Performance Testing Tool
Version 2.14.0

[Status] Command line: dnsperf -f any -m udp -s 10.96.0.10 -p 53 -d /opt/records.txt -c 1 -T 1 -l 30 -t 5 -Q 100000
[Status] Sending queries (to 10.96.0.10:53)
[Status] Started at: Sun Jan 12 20:45:41 2025
[Status] Stopping after 30.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         387037
  Queries completed:    387037 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 387037 (100.00%)
  Average packet size:  request 43, response 67
  Run time (s):         30.006926
  Queries per second:   12898.255556

  Average Latency (s):  0.007742 (min 0.000133, max 0.013404)
  Latency StdDev (s):   0.000808

DNS Performance Testing Tool
Version 2.14.0

DNS Performance Testing Tool
Version 2.14.0

[Status] Command line: dnsperf -f any -m udp -s 10.96.0.10 -p 53 -d /opt/records.txt -c 1 -T 1 -l 30 -t 5 -Q 100000
[Status] Sending queries (to 10.96.0.10:53)
[Status] Started at: Sun Jan 12 20:49:54 2025
[Status] Stopping after 30.000000 seconds
[Status] Testing complete (time limit)

Statistics:

  Queries sent:         2203514
  Queries completed:    2203514 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 2203514 (100.00%)
  Average packet size:  request 43, response 160
  Run time (s):         30.007927
  Queries per second:   73431.063732

  Average Latency (s):  0.001351 (min 0.000025, max 0.042577)
  Latency StdDev (s):   0.001283

DNS Performance Testing Tool
Version 2.14.0

With cache enabled, the min latency is faster without the cache, it seems I need to

Independently, now each DNS request does NOT generate any UDP conntrack entry, a TCP connection is created and reused in a pool, with a maximum of 3 TCP connections per Node.
Before a name resolution generated one request per family (A and AAAA), and one request for dns search domain in the resolv.conf, now it can generate maximum one request, the rest are pipelined through the TCP connection., cc: @thockin

TODO:

explore https://datatracker.ietf.org/doc/html/rfc7828#section-3
optimize the fast path to make the cache

justinsb reviewed Nov 5, 2024

View reviewed changes

aojea force-pushed the magicdns branch 12 times, most recently from 4e5d358 to 7cc54d6 Compare January 12, 2025 19:54

aojea added 9 commits January 12, 2025 20:07

ci: chmod only downloaded binaries

d0c695f

move packet code to network and add unit tests for nflog

13d0fd3

cni more resilient to kernel netlink races

09ed9d2

add unit tests for fastpath

ffebf01

add permision for read kubelet local endpoint

f8135a6

export packet fields

d2f9452

distroless

a28ddc0

connection pool

8591ac4

dns cache and tcp pipelining

9e9dfaa

enable dns cache by default, it has the proper guardrails to avoid to break clusters, if anything fails processing the packet it falls back to the dataplane.

aojea force-pushed the magicdns branch from 7cc54d6 to 9e9dfaa Compare January 12, 2025 20:24

aojea merged commit 3c8c2cd into main Jan 12, 2025
6 checks passed

aojea mentioned this pull request Jan 12, 2025

Seeking Contributors for Tutorials and Documentation #147

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Magicdns #79

Magicdns #79

aojea commented Oct 27, 2024

justinsb Nov 5, 2024

justinsb Nov 5, 2024

justinsb Nov 5, 2024

aojea Nov 5, 2024

justinsb Nov 5, 2024

aojea Nov 5, 2024

justinsb Nov 5, 2024

aojea Nov 5, 2024

justinsb Nov 5, 2024

justinsb Nov 5, 2024

justinsb commented Nov 5, 2024

aojea commented Dec 24, 2024 •

edited

Loading

aojea commented Jan 10, 2025

aojea commented Jan 12, 2025

Magicdns #79

Magicdns #79

Conversation

aojea commented Oct 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb commented Nov 5, 2024

aojea commented Dec 24, 2024 • edited Loading

aojea commented Jan 10, 2025

aojea commented Jan 12, 2025

aojea commented Dec 24, 2024 •

edited

Loading