Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flannel.1 cannot forward the packets to docker0 #731

Closed
KDF5000 opened this issue May 19, 2017 · 12 comments
Closed

Flannel.1 cannot forward the packets to docker0 #731

KDF5000 opened this issue May 19, 2017 · 12 comments

Comments

@KDF5000
Copy link

KDF5000 commented May 19, 2017

Hi,
I deployed k8s cluster with two nodes using Flannel as the overlay network. The deployment env is:
OS: Ubuntu 14.04
Kubernetes: 1.6.2
Etcd: 3.1.5
Flannel: 0.7.1

One physical node(A) is as master and node , while another(B) is just node. Then I create a pod on each node. I can ping the pod on the Master node(A) from B successfully, but cannot get response when ping the pod on B from A. So I trace the iptables using target TRACE, and got the following messaege:

May 19 19:52:12 gd87 kernel: [11137779.583962] TRACE: raw:PREROUTING:policy:2 IN=flannel.1 OUT= MAC=b6:9e:8b:64:df:20:4e:00:65:87:3a:15:08:00:45:00:00:54:fb:62:40:00:40:01:46:1b:ac:14:61:00:ac:14:40:02:08:00:b5:ff:4e:50:00:de:c2:dc:1e:59:00:00:00:00:4d:c9:05:00:00:00:00:00:10:11:12:13:14:15 SRC=172.20.97.0 DST=172.20.64.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=64354 DF PROTO=ICMP TYPE=8 CODE=0 ID=20048 SEQ=222 
May 19 19:52:12 gd87 kernel: [11137779.583989] TRACE: filter:INPUT:rule:1 IN=flannel.1 OUT= MAC=b6:9e:8b:64:df:20:4e:00:65:87:3a:15:08:00:45:00:00:54:fb:62:40:00:40:01:46:1b:ac:14:61:00:ac:14:40:02:08:00:b5:ff:4e:50:00:de:c2:dc:1e:59:00:00:00:00:4d:c9:05:00:00:00:00:00:10:11:12:13:14:15 SRC=172.20.97.0 DST=172.20.64.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=64354 DF PROTO=ICMP TYPE=8 CODE=0 ID=20048 SEQ=222 
May 19 19:52:12 gd87 kernel: [11137779.584009] TRACE: filter:KUBE-FIREWALL:return:2 IN=flannel.1 OUT= MAC=b6:9e:8b:64:df:20:4e:00:65:87:3a:15:08:00:45:00:00:54:fb:62:40:00:40:01:46:1b:ac:14:61:00:ac:14:40:02:08:00:b5:ff:4e:50:00:de:c2:dc:1e:59:00:00:00:00:4d:c9:05:00:00:00:00:00:10:11:12:13:14:15 SRC=172.20.97.0 DST=172.20.64.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=64354 DF PROTO=ICMP TYPE=8 CODE=0 ID=20048 SEQ=222 
May 19 19:52:12 gd87 kernel: [11137779.584027] TRACE: filter:INPUT:policy:2 IN=flannel.1 OUT= MAC=b6:9e:8b:64:df:20:4e:00:65:87:3a:15:08:00:45:00:00:54:fb:62:40:00:40:01:46:1b:ac:14:61:00:ac:14:40:02:08:00:b5:ff:4e:50:00:de:c2:dc:1e:59:00:00:00:00:4d:c9:05:00:00:00:00:00:10:11:12:13:14:15 SRC=172.20.97.0 DST=172.20.64.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=64354 DF PROTO=ICMP TYPE=8 CODE=0 ID=20048 SEQ=222

the mac address is weird. When ping from A to B, the log is as the following:

May 19 20:32:23 gd86 kernel: [890898.586760] TRACE: raw:PREROUTING:policy:2 IN=flannel.1 OUT= MAC=4e:00:65:87:3a:15:b6:9e:8b:64:df:20:08:00 SRC=172.20.64.0 DST=172.20.97.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53616 DF PROTO=ICMP TYPE=8 CODE=0 ID=15769 SEQ=7 
May 19 20:32:23 gd86 kernel: [890898.586797] TRACE: filter:FORWARD:rule:1 IN=flannel.1 OUT=docker0 MAC=4e:00:65:87:3a:15:b6:9e:8b:64:df:20:08:00 SRC=172.20.64.0 DST=172.20.97.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53616 DF PROTO=ICMP TYPE=8 CODE=0 ID=15769 SEQ=7 
May 19 20:32:23 gd86 kernel: [890898.586810] TRACE: filter:DOCKER-ISOLATION:return:1 IN=flannel.1 OUT=docker0 MAC=4e:00:65:87:3a:15:b6:9e:8b:64:df:20:08:00 SRC=172.20.64.0 DST=172.20.97.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53616 DF PROTO=ICMP TYPE=8 CODE=0 ID=15769 SEQ=7 
May 19 20:32:23 gd86 kernel: [890898.586820] TRACE: filter:FORWARD:rule:2 IN=flannel.1 OUT=docker0 MAC=4e:00:65:87:3a:15:b6:9e:8b:64:df:20:08:00 SRC=172.20.64.0 DST=172.20.97.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53616 DF PROTO=ICMP TYPE=8 CODE=0 ID=15769 SEQ=7 
May 19 20:32:23 gd86 kernel: [890898.586831] TRACE: filter:DOCKER:return:1 IN=flannel.1 OUT=docker0 MAC=4e:00:65:87:3a:15:b6:9e:8b:64:df:20:08:00 SRC=172.20.64.0 DST=172.20.97.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53616 DF PROTO=ICMP TYPE=8 CODE=0 ID=15769 SEQ=7 
May 19 20:32:23 gd86 kernel: [890898.586843] TRACE: filter:FORWARD:rule:3 IN=flannel.1 OUT=docker0 MAC=4e:00:65:87:3a:15:b6:9e:8b:64:df:20:08:00 SRC=172.20.64.0 DST=172.20.97.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53616 DF PROTO=ICMP TYPE=8 CODE=0 ID=15769 SEQ=7

It seems that the iptables drop the packets from flannel.1 to docker0, but I have no idea that how to solve it. I compare the iptables rules on the two nodes. I didn`t find any difference.

And the route info is different with the one using flannel 0.5.5:
flannel 0.7.1:

172.16.0.0/16 dev eth0  proto kernel  scope link  src 172.16.1.86 
172.20.0.0/16 dev flannel.1 
172.20.97.0/24 dev docker0  proto kernel  scope link  src 172.20.97.1

flannel 0.5.5:

172.16.0.0/16 dev eth0  proto kernel  scope link  src 172.16.1.86 
172.20.0.0/16 dev flannel.1 proto kernel  scope link  src 172.20.97.0
172.20.97.0/24 dev docker0  proto kernel  scope link  src 172.20.97.1

I don`t know whether there are some bugs with flannel 0.7.1. So what can I do to solve the problem? Thanks !
@whybeyoung
Copy link

whybeyoung commented May 20, 2017 •
edited
Loading

i meet the same problems wit you. can't forward packets to docker0,my issue：kubernetes/kubernetes#46077
my master is ubunu 16.04
and all the nodes is 14.04
node's docker0 can ping success but the pod some time ok,some time bad.
i see that you use 14.04 too, my 14.04 node kernel version is Linux i-40C8C8F5 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
how about you?

@defcyy
Copy link

defcyy commented May 20, 2017

@KDF5000 the problem seems to be the same as #609 , sudo iptables -P FORWARD ACCEPT works for me.

@whybeyoung
Copy link

whybeyoung commented May 21, 2017 •
edited
Loading

@defcyy i don't know how @KDF5000 did, i did the sudo iptables -P FORWARD ACCEPT ,
but not use, i make success sometimes , but when i delete the pods ,and after the pods rebuilt, the same issues became.

@whybeyoung
Copy link

whybeyoung commented May 21, 2017 •
edited
Loading

i guess it's the ubuntu 14.04 kernel's problem;
Linux i-40C8C8F5 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
i take tests for many times; sometimes ok ,but usually has ping problem between the different node's pods

@KDF5000
Copy link
Author

KDF5000 commented May 21, 2017

@berlinsaint @defcyy my linux kernel version is the same as yours. I added the forward rule "iptables -P FORWARD ACCEPT" ,but it still did not work. The flannel version you use is 0.7.1?

@whybeyoung
Copy link

whybeyoung commented May 21, 2017 •
edited
Loading

@KDF5000 071 too, i guess your kernel version is 3.13 right? i update my kernel ,the problem solved.
i update to Linux i-40C8C8F5 4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux then every thing becomes ok.

@KDF5000
Copy link
Author

KDF5000 commented May 21, 2017

@berlinsaint yes, it is 3.13.0-24. Did you upgrade to 16.04 or rebuild a new kernel?

@whybeyoung
Copy link

nop, just upgrade the kernel. use
sudo apt-get install linux-generic-lts-wily

@whybeyoung
Copy link

靠，都是中国人，QQ 66375364 一起交流

@whybeyoung
Copy link

你要是 升级内核后也解决问题了，我建议我们提一个issue，官方flannel貌似没有说 有内核限制吧？我好像没看到。

@KDF5000
Copy link
Author

KDF5000 commented May 21, 2017

@berlinsaint @defcyy After I upgrade the kernel to 4.2.0-42, it works! Thanks!! @berlinsaint @defcyy

There are some related issues which will be helpful:
moby/moby#28257
Support VXLAN on Kernels <= 3.15

ps: @berlinsaint 你的qq需要验证[捂脸]

@whybeyoung
Copy link

我发邮件给你， 你回复我下 @KDF5000

@KDF5000 KDF5000 closed this as completed May 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@defcyy @KDF5000 @whybeyoung