DNS broken on Ubuntu 15.10 desktop (w UFW) #1578

inercia · 2015-10-23T11:04:01Z

When running Weave on Ubuntu:

$ ./weave launch
$ C2=$(./weave run --name c1 -ti ubuntu  /bin/bash)
$ C2=$(./weave run --name c2 -ti ubuntu  /bin/bash)
$ docker attach $C2
# ping c1
ping: unknown host c1

The container can ping the DNS resolver:

# cat /etc/resolv.conf 
nameserver 172.17.42.1
# ping 172.17.42.1
PING 172.17.42.1 (172.17.42.1) 56(84) bytes of data.
64 bytes from 172.17.42.1: icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 172.17.42.1: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 172.17.42.1: icmp_seq=3 ttl=64 time=0.059 ms
64 bytes from 172.17.42.1: icmp_seq=4 ttl=64 time=0.059 ms
...

But ufw seems to be blocking traffic to the DNS server:

$ sudo tcpdump -n -i docker0    "port 53"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on docker0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:59:08.381052 IP 172.17.0.2.48960 > 172.17.42.1.53: 56666+ A? c1.weave.local. (32)
12:59:13.381821 IP 172.17.0.2.48960 > 172.17.42.1.53: 56666+ A? c1.weave.local. (32)
12:59:18.383369 IP 172.17.0.2.37360 > 172.17.42.1.53: 30668+ A? c1. (20)
12:59:23.388212 IP 172.17.0.2.37360 > 172.17.42.1.53: 30668+ A? c1. (20)

It can be seen with

$ dmesg 
[ 3029.623661] [UFW BLOCK] IN=docker0 OUT= PHYSIN=vethf203be5 MAC=02:42:b6:97:66:4f:02:42:ac:11:00:02:08:00 SRC=172.17.0.2 DST=172.17.42.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52288 DF PROTO=UDP SPT=54743 DPT=53 LEN=40

The text was updated successfully, but these errors were encountered:

inercia · 2015-10-23T11:20:04Z

It can be fixed with

sudo iptables -t filter -I INPUT -i docker0 -p udp --dport 53  -j ACCEPT

rade · 2015-10-23T15:27:24Z

Please check that weave expose works ok, just in case ufw is throwing a spanner in the works there too.

rade · 2015-10-28T11:13:10Z

what version of ubuntu is this problem occurring in?

inercia · 2015-10-28T12:46:33Z

what version of ubuntu is this problem occurring in?

I'm using Ubuntu 15.10

rade · 2015-10-30T01:34:25Z

I have just run the example in a ubuntu 15.10 vagrant VM and it works fine.

So perhaps the problem is confined to 15.10 desktop VMs.

Please post the output of iptables-save.

rade · 2015-10-30T01:36:58Z

also, what docker version are you running?

rade · 2015-10-30T01:48:04Z

I've just run our entire test suite against 15.10 vagrant VMs, and it passed.

inercia · 2015-11-02T10:03:50Z

$ docker version
Client:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:39:44 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.3
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   f4bf5c7
 Built:        Mon Oct 12 05:39:44 UTC 2015
 OS/Arch:      linux/amd64

$ sudo iptables-save
# Generated by iptables-save v1.4.21 on Mon Nov  2 11:01:28 2015
*mangle
:PREROUTING ACCEPT [1371107:1679981379]
:INPUT ACCEPT [1339221:1678955447]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1215710:386118054]
:POSTROUTING ACCEPT [1219584:386535982]
-A INPUT -p esp -j MARK --set-xmark 0x1/0x1
-A INPUT -p udp -m udp --dport 4500 -j MARK --set-xmark 0x1/0x1
COMMIT
# Completed on Mon Nov  2 11:01:28 2015
# Generated by iptables-save v1.4.21 on Mon Nov  2 11:01:28 2015
*nat
:PREROUTING ACCEPT [88248:4250191]
:INPUT ACCEPT [113:9403]
:OUTPUT ACCEPT [38666:7163444]
:POSTROUTING ACCEPT [38370:7142612]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
COMMIT
# Completed on Mon Nov  2 11:01:28 2015
# Generated by iptables-save v1.4.21 on Mon Nov  2 11:01:28 2015
*filter
:INPUT DROP [10560:348192]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [439:22696]
:DOCKER - [0:0]
:ufw-after-forward - [0:0]
:ufw-after-input - [0:0]
:ufw-after-logging-forward - [0:0]
:ufw-after-logging-input - [0:0]
:ufw-after-logging-output - [0:0]
:ufw-after-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-before-input - [0:0]
:ufw-before-logging-forward - [0:0]
:ufw-before-logging-input - [0:0]
:ufw-before-logging-output - [0:0]
:ufw-before-output - [0:0]
:ufw-logging-allow - [0:0]
:ufw-logging-deny - [0:0]
:ufw-not-local - [0:0]
:ufw-reject-forward - [0:0]
:ufw-reject-input - [0:0]
:ufw-reject-output - [0:0]
:ufw-skip-to-policy-forward - [0:0]
:ufw-skip-to-policy-input - [0:0]
:ufw-skip-to-policy-output - [0:0]
:ufw-track-forward - [0:0]
:ufw-track-input - [0:0]
:ufw-track-output - [0:0]
:ufw-user-forward - [0:0]
:ufw-user-input - [0:0]
:ufw-user-limit - [0:0]
:ufw-user-limit-accept - [0:0]
:ufw-user-logging-forward - [0:0]
:ufw-user-logging-input - [0:0]
:ufw-user-logging-output - [0:0]
:ufw-user-output - [0:0]
-A INPUT -j ufw-before-logging-input
-A INPUT -j ufw-before-input
-A INPUT -j ufw-after-input
-A INPUT -j ufw-after-logging-input
-A INPUT -j ufw-reject-input
-A INPUT -j ufw-track-input
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
-A FORWARD -j ufw-after-forward
-A FORWARD -j ufw-after-logging-forward
-A FORWARD -j ufw-reject-forward
-A FORWARD -j ufw-track-forward
-A OUTPUT -j ufw-before-logging-output
-A OUTPUT -j ufw-before-output
-A OUTPUT -j ufw-after-output
-A OUTPUT -j ufw-after-logging-output
-A OUTPUT -j ufw-reject-output
-A OUTPUT -j ufw-track-output
-A ufw-after-input -p udp -m udp --dport 137 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 138 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 139 -j ufw-skip-to-policy-input
-A ufw-after-input -p tcp -m tcp --dport 445 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 67 -j ufw-skip-to-policy-input
-A ufw-after-input -p udp -m udp --dport 68 -j ufw-skip-to-policy-input
-A ufw-after-input -m addrtype --dst-type BROADCAST -j ufw-skip-to-policy-input
-A ufw-after-logging-input -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-before-forward -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-forward -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-forward -j ufw-user-forward
-A ufw-before-input -i lo -j ACCEPT
-A ufw-before-input -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-input -m conntrack --ctstate INVALID -j ufw-logging-deny
-A ufw-before-input -m conntrack --ctstate INVALID -j DROP
-A ufw-before-input -p icmp -m icmp --icmp-type 3 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 4 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 11 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 12 -j ACCEPT
-A ufw-before-input -p icmp -m icmp --icmp-type 8 -j ACCEPT
-A ufw-before-input -p udp -m udp --sport 67 --dport 68 -j ACCEPT
-A ufw-before-input -j ufw-not-local
-A ufw-before-input -d 224.0.0.251/32 -p udp -m udp --dport 5353 -j ACCEPT
-A ufw-before-input -d 239.255.255.250/32 -p udp -m udp --dport 1900 -j ACCEPT
-A ufw-before-input -j ufw-user-input
-A ufw-before-output -o lo -j ACCEPT
-A ufw-before-output -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -j ufw-user-output
-A ufw-logging-allow -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW ALLOW] "
-A ufw-logging-deny -m conntrack --ctstate INVALID -m limit --limit 3/min --limit-burst 10 -j RETURN
-A ufw-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW BLOCK] "
-A ufw-not-local -m addrtype --dst-type LOCAL -j RETURN
-A ufw-not-local -m addrtype --dst-type MULTICAST -j RETURN
-A ufw-not-local -m addrtype --dst-type BROADCAST -j RETURN
-A ufw-not-local -m limit --limit 3/min --limit-burst 10 -j ufw-logging-deny
-A ufw-not-local -j DROP
-A ufw-skip-to-policy-forward -j ACCEPT
-A ufw-skip-to-policy-input -j DROP
-A ufw-skip-to-policy-output -j ACCEPT
-A ufw-track-forward -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-forward -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p tcp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-track-output -p udp -m conntrack --ctstate NEW -j ACCEPT
-A ufw-user-input -p tcp -m tcp --dport 2375 -j ACCEPT
-A ufw-user-limit -m limit --limit 3/min -j LOG --log-prefix "[UFW LIMIT BLOCK] "
-A ufw-user-limit -j REJECT --reject-with icmp-port-unreachable
-A ufw-user-limit-accept -j ACCEPT
COMMIT
# Completed on Mon Nov  2 11:01:28 2015

rade · 2015-11-02T11:24:53Z

So it turns out ufw wasn't enabled in my vagrant VMs.

Here's how I got this all to work in the end:

apply this patch:

diff --git a/test/Vagrantfile b/test/Vagrantfile
index 1080bfd..eca6475 100644
--- a/test/Vagrantfile
+++ b/test/Vagrantfile
@@ -14,8 +14,7 @@ require '../vagrant-common.rb'
def configure_docker(host, hostname, ip)
 pkgs = %w(docker-engine ethtool)

-  host.vm.box = "ubuntu/ubuntu-15.04-amd64"
-  host.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/vivid/current/vivid-server-cloudimg-amd64-vagrant-disk1.box"
+  host.vm.box = "ubuntu/wily64"

host.vm.provision :shell, :inline => "hostnamectl set-hostname "+hostname
host.vm.network "private_network", ip: ip
@@ -23,7 +22,7 @@ def configure_docker(host, hostname, ip)
host.vm.synced_folder ".", "/vagrant", disabled: true

host.vm.provision :shell, :inline => "sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D"
-  host.vm.provision :shell, :inline => "echo deb https://apt.dockerproject.org/repo ubuntu-vivid main > /etc/apt/sources.list.d/docker.list"
+  host.vm.provision :shell, :inline => "echo deb https://apt.dockerproject.org/repo ubuntu-wily main > /etc/apt/sources.list.d/docker.list"

install_packages host.vm, pkgs
tweak_docker_daemon host.vm

run vagrant up
vagrant ssh into each VM and run:

$ sudo ufw enable
$ sudo ufw allow ssh
$ sudo ufw allow 2375/tcp
$ sudo ufw allow 12375/tcp
$ sudo ufw allow 6783

awh · 2015-11-04T15:42:57Z

Closed by 781c805

rade added the chore label Oct 23, 2015

rade added this to the 1.2.1 milestone Oct 23, 2015

rade added the [component/dns] label Oct 23, 2015

inercia self-assigned this Oct 23, 2015

inercia mentioned this issue Oct 23, 2015

Add an iptables rule for letting weaveDNS requests in #1581

Closed

inercia mentioned this issue Oct 28, 2015

Add an iptables rule for letting weaveDNS requests in #1606

Merged

rade changed the title ~~Make Weave play nicely with UFW on Ubuntu~~ DNS broken on Ubuntu 15.10 desktop (w UFW) Nov 3, 2015

rade added bug and removed chore labels Nov 3, 2015

awh closed this as completed Nov 4, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DNS broken on Ubuntu 15.10 desktop (w UFW) #1578

DNS broken on Ubuntu 15.10 desktop (w UFW) #1578

inercia commented Oct 23, 2015

inercia commented Oct 23, 2015

rade commented Oct 23, 2015

rade commented Oct 28, 2015

inercia commented Oct 28, 2015

rade commented Oct 30, 2015

rade commented Oct 30, 2015

rade commented Oct 30, 2015

inercia commented Nov 2, 2015

rade commented Nov 2, 2015

awh commented Nov 4, 2015

DNS broken on Ubuntu 15.10 desktop (w UFW) #1578

DNS broken on Ubuntu 15.10 desktop (w UFW) #1578

Comments

inercia commented Oct 23, 2015

inercia commented Oct 23, 2015

rade commented Oct 23, 2015

rade commented Oct 28, 2015

inercia commented Oct 28, 2015

rade commented Oct 30, 2015

rade commented Oct 30, 2015

rade commented Oct 30, 2015

inercia commented Nov 2, 2015

rade commented Nov 2, 2015

awh commented Nov 4, 2015