Weave restarting after a few hours #2768

chrislovecnm · 2017-02-03T04:58:35Z

Hi all

We are having weave-npc restarting on some nodes after about 10 - 12 hrs. We are running 1.8.2 this is an error message I was able to dig up:

$ kubectl -n kube-system logs -p weave-net-xq8ia weave
2017/02/02 20:27:27 error contacting APIServer: Get https://100.64.0.1:443/api/v1/nodes: dial tcp 100.64.0.1:443: i/o timeout; trying with fallback: http://localhost:8080
2017/02/02 20:27:27 Could not get peers: Get http://localhost:8080/api/v1/nodes: dial TCP [::1]:8080: getsockopt: connection refused
Failed to get peers

kube-dns has been failing with UDP network connection errors, and a Zookeeper member was in CrashLoop because of the network. We have cordoned the nodes off and would like to diagnose the problem.

What steps should we go through to diagnose the problem?
What weave commands can we run on the nodes?
What tools besides tcpdump do you usually use?

Working on collecting more information on this issue.

The text was updated successfully, but these errors were encountered:

chrislovecnm · 2017-02-03T05:25:16Z

Interesting chatter in dmesg:

95874.683839] weave: port 4(vethweplc52849e) entered forwarding state
[95874.688118] IPv6: eth0: IPv6 duplicate address fe80::7883:89ff:feb5:4ccc detected!
[95874.691647] IPv6: ADDRCONF(NETDEV_CHANGE): vethweplc52849e: link becomes ready
[95875.152152] IPv6: eth0: IPv6 duplicate address fe80::f45c:f2ff:fe1a:a25c detected!
[95875.302369] weave: port 16(vethweplc232b92) entered disabled state
[95875.308866] device vethweplc232b92 left promiscuous mode
[95875.311595] weave: port 16(vethweplc232b92) entered disabled state
[95879.315840] weave: port 12(vethwepl3a008bd) entered disabled state
....
[96037.144080] unregister_netdevice: waiting for lo to become free. Usage count = 1

Question

do we need to disable ipv6?

I think I have seen this error by kubelet and it is not an error:

72-20-79-243 kubelet[1278]: E0201 17:57:10.570951    1278 cni.go:163] error updating cni config: No networks found in /etc/cni/net.d

This is an odd message

Feb  1 18:12:54 ip-172-20-79-243 kubelet[1278]: E0201 18:12:54.376221    1278 docker_manager.go:357] NetworkPlugin cni failed on the status hook for pod 'zookeeper-0' - Unexpected command output Device "eth0" does not exist.

Seeing the double remove errors where cni is trying to remove the network after it has already been removed.

Have not seen these errors before:

eb  2 03:40:07 ip-172-20-79-243 kubelet[1278]: E0202 03:40:07.638036    1278 cni.go:273] Error deleting network: unable to release IP address: dial tcp 127.0.0.1:6784: getsockopt: connection refused
Feb  2 03:40:07 ip-172-20-79-243 kubelet[1278]: E0202 03:40:07.638064    1278 docker_manager.go:1421] Failed to teardown network for pod "7003617f-e8a8-11e6-ae21-0ae7c18cf079" using network plugins "cni": unable to release IP address: dial tcp 127.0.0.1:6784: getsockopt: connection refused

Specifically weave-npc is restarting - and I have zero logs on weave-npc on 243.

Kernel and ethtool

root@ip-172-20-79-243:~# which ethtool
/sbin/ethtool
root@ip-172-20-79-243:~# uname -a
Linux ip-172-20-79-243 4.4.41-k8s #1 SMP Mon Jan 9 15:34:39 UTC 2017 x86_64 GNU/Linux

chrislovecnm · 2017-02-03T05:48:20Z

I only have logging in one weave-npc pod, and I am getting:

time="2017-02-03T05:46:44Z" level=warning msg="TCP connection from 10.46.0.1:36504 to 172.20.35.34:10255 blocked by Weave NPC."

chrislovecnm · 2017-02-03T05:55:55Z

cc @awh @bboreham @scottvidmar

chrislovecnm · 2017-02-03T06:00:18Z

Another question kubernetes/kops#1171 should we be using nonMasqueradeCIDR for the weave subnet block?

marccarre · 2017-02-03T11:59:52Z

Hi Chris,

The below errors basically mean that Weave cannot reach Kubernetes, so let's proceed step by step, backwards from this information.

$ kubectl -n kube-system logs -p weave-net-xq8ia weave
2017/02/02 20:27:27 error contacting APIServer: Get https://100.64.0.1:443/api/v1/nodes: dial tcp 100.64.0.1:443: i/o timeout; trying with fallback: http://localhost:8080
2017/02/02 20:27:27 Could not get peers: Get http://localhost:8080/api/v1/nodes: dial TCP [::1]:8080: getsockopt: connection refused
Failed to get peers

Given the above was logged by Weave's kube-peers process, which lives in the host namespace, it is fair to assume the issue is related to either:

the host network, or
kube-proxy

Steps:

What steps should we go through to diagnose the problem?

check if iptables has a mapping for Kubernetes' API server (100.64.0.1:443), and if so,
check why the HTTPS call timed out.

Detailed steps:

check if iptables has a mapping for Kubernetes' API server (100.64.0.1:443).
kube-proxy creates iptables rules that rewrite the destination of an IP packet sent to any service. To check the iptables rules (see also here):
1. run sudo iptables-save on the host -- that will give you all the rules;
2. look for the service IP/port you are interested in -- here: 100.64.0.1 and 443;
3. at the end of the line it jumps to another rule (e.g. -j KUBE-SVC-XXX), find that rule;
4. the next rule, is generally just another indirection (e.g. -j KUBE-SEP-XXX), find that rule;
5. this rule maps to zero or more IP addresses:
  1. if it does not map to any IP address, then the machine will attempt to send the packet to 100.64.0.1, which is unlikely to be a real IP address, and therefore the packet will get dropped;
  2. if it maps to some IP addresses, then:
check why the HTTPS call timed out
Given the above error (or rather because it didn't give different errors), what we can say at the moment is:
1. the machine thought it could reach that address,
2. it didn't reach an interface that rejected the connection.

In any case, let's first get your reply regarding the above.
I'll address other questions later on.

chrislovecnm · 2017-02-03T15:39:41Z

check if iptables has a mapping for Kubernetes' API server (100.64.0.1:443), and if so,
check why the HTTPS call timed out.

None, from ip-172-20-79-243 - it can curl the API server. Checking the other node that is cordoned. The other node does not have any communication problems to the master either.

check why the HTTPS call timed out.

It does not timeout now.

Does not now, and we had another weave pod restart last night.

What are the weave diagnostic commands for weave-net?

chrislovecnm · 2017-02-03T15:51:03Z

Did not mention the k8s version:

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.8", GitCommit:"c5fcb1951afb8150e4574bc234aed957cb586eb0", GitTreeState:"clean", BuildDate:"2017-01-12T02:14:29Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

chrislovecnm · 2017-02-03T18:01:22Z

One of the commands to debug weave is weave status - with the ds we only have the weave-net binary.

bboreham · 2017-02-03T18:24:51Z

Two options:

Install the weave command per the usual instructions (but don't run weave launch)
Hit the internal api directly: curl http://127.0.0.1:6784/status (or /status/connections, etc)

chrislovecnm · 2017-02-04T15:12:23Z

So yesterday at 17:47 MT

weave-net-0auok                                                       2/2       Running   11         2d
weave-net-6af9j                                                       2/2       Running   0          6h
weave-net-ciyk4                                                       2/2       Running   11         2d
weave-net-i4hnp                                                       2/2       Running   1          10h
weave-net-jxx1i                                                       2/2       Running   1          1d
weave-net-tyn03                                                       2/2       Running   11         2d
weave-net-wuthw                                                       2/2       Running   1          1d
weave-net-xq8ia                                                       2/2       Running   2          1d

And this morning 7:54:29 MT

weave-net-0auok                                                       2/2       Running   11         2d
weave-net-2mphq                                                       2/2       Running   0          2h
weave-net-6af9j                                                       2/2       Running   0          20h
weave-net-ciyk4                                                       2/2       Running   11         2d
weave-net-jxx1i                                                       2/2       Running   1          1d
weave-net-tyn03                                                       2/2       Running   11         2d
weave-net-wuthw                                                       2/2       Running   1          2d
weave-net-y6mru                                                       2/2       Running   0          8h

And we lost another node ip-172-20-81-117.us-west-2.compute.internal - which is running weave-net-y6mru .

I am able to get the logs for weave-net-y6mru weave but not weave-net-y6mru weave-net - I think the weave-net container are not starting correctly, and the weave container is. We have seen this in 1.8.x before. Thoughts?

also @bboreham this is a shot in the dark but could this be impacting this kubernetes/kops#1171

bboreham · 2017-02-04T15:22:26Z

kops#1171 is about reconfiguring the IP address range? I can't see why that would be relevant.

Weave peers all need to be configured with the same IP range, but there is a specific error message if they are mismatched, and you're not getting that message.

chrislovecnm · 2017-02-04T18:53:56Z

With help I am getting more logs:

INFO: 2017/02/04 02:50:43.790561 ->[172.20.111.53:6783] error during connection attempt: dial tcp4 :0->172.20.111.53:6783: getsockopt: connection timed out

INFO: 2017/02/04 04:35:46.497130 ->[172.20.81.117:38452|5a:a4:1d:20:42:c0(ip-172-20-81-117)]: connection shutting down due to error: cannot connect to ourself

chrislovecnm · 2017-02-04T19:03:22Z

Any idea of some keywords that I can search for?

chrislovecnm · 2017-02-04T19:40:33Z

@bboreham you asked about hairpin

Feb  1 17:55:18 ip-172-20-35-34 kubelet[1265]: W0201 17:55:18.401759    1265 kubelet_network.go:71] Hairpin mode set to "promiscuous-bridge" but configureCBR0 is false, falling back to "hairpin-veth"
Feb  1 17:55:18 ip-172-20-35-34 kubelet[1265]: I0201 17:55:18.401782    1265 kubelet.go:513] Hairpin mode set to "hairpin-veth"

chrislovecnm · 2017-02-04T19:41:27Z

Nothing is standing out to me in the logs ... hopefully we can dig around in kibana when we talk

bboreham · 2017-02-04T20:23:47Z

keywords

"Stack dump" or just "dump" would indicate weave net had crashed.

Or just the end of the log before a restart.

In the kubelet logs I would be interests in what happened around the time the errors start (06:57), or any errors after the first minute or so when you can get some noise due to things not being ready yet.

jroberts235 · 2017-02-05T02:06:39Z

On ip-172-20-81-117: Tried setting some kernel params to see if it would have any positive effects on this issue (pods are failing suddenly on this cluster):

echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 15000 65000 > /proc/sys/net/ipv4/ip_local_port_range
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
echo 1 > cat /proc/sys/net/ipv4/tcp_tw_reuse

I restarted all of the weave pods on the cluster. Pods worked again for about 5 minutes, then began to fail again.

The number of sockets in either TIME_WAIT or CLOSE_WAIT was never high (<64 in total) but after the kernel params were tuned, TIME_WAIT went to 0 and only 37 sockets remained in CLOSE_WAIT.

The number of sockets, in some WAIT state, on the other nodes in the cluster were much higher:

ip-172-20-100-20.us-west-2.compute.internal: 422
ip-172-20-101-115.us-west-2.compute.internal: 7
ip-172-20-35-34.us-west-2.compute.internal: 142
ip-172-20-42-174.us-west-2.compute.internal: 75
ip-172-20-63-25.us-west-2.compute.internal: 392
ip-172-20-79-243.us-west-2.compute.internal: 142
ip-172-20-81-117.us-west-2.compute.internal: 37
ip-172-20-91-60.us-west-2.compute.internal: 373

I curl'd the weave status endpoint and got the following results:

root@ip-172-20-81-117:~# curl http://127.0.0.1:6784/status
Version: 1.8.2 (up to date; next check at 2017/02/05 08:50:41)

    Service: router
   Protocol: weave 1..2
       Name: 5a:a4:1d:20:42:c0(ip-172-20-81-117)
 Encryption: disabled

PeerDiscovery: enabled
Targets: 8
Connections: 8 (7 established, 1 failed)
Peers: 8 (with 56 established connections)
TrustedSubnets: none

    Service: ipam
     Status: ready
      Range: 10.32.0.0/12

DefaultSubnet: 10.32.0.0/12
root@ip-172-20-81-117:~# curl http://127.0.0.1:6784/status/connections
-> 172.20.91.60:6783 established fastdp ba:b4:a9:76:9a:e5(ip-172-20-91-60)
-> 172.20.35.34:6783 established fastdp 1a:cc:23:8a:b4:f1(ip-172-20-35-34)
-> 172.20.101.115:6783 established fastdp ce:cf:f9:61:89:0b(ip-172-20-101-115)
<- 172.20.63.25:50844 established fastdp 76:fc:72:a9:29:9e(ip-172-20-63-25)
<- 172.20.79.243:56543 established fastdp 9a:74:7a:fd:4b:44(ip-172-20-79-243)
<- 172.20.100.20:48399 established fastdp 7a:8c:e3:aa:36:c3(ip-172-20-100-20)
<- 172.20.42.174:57333 established fastdp 86:4e:8b:7e:79:59(ip-172-20-42-174)
-> 172.20.81.117:6783 failed cannot connect to ourself, retry: never

Why can weave talk to itself???

I set the same kernel params on each node in the cluster:

after 1 min:

ip-172-20-100-20.us-west-2.compute.internal: 296
ip-172-20-101-115.us-west-2.compute.internal: 1
ip-172-20-35-34.us-west-2.compute.internal: 136
ip-172-20-42-174.us-west-2.compute.internal: 48
ip-172-20-63-25.us-west-2.compute.internal: 282
ip-172-20-79-243.us-west-2.compute.internal: 138
ip-172-20-81-117.us-west-2.compute.internal: 37
ip-172-20-91-60.us-west-2.compute.internal: 341

after 3 min:

ip-172-20-100-20.us-west-2.compute.internal: 288
ip-172-20-101-115.us-west-2.compute.internal: 1
ip-172-20-35-34.us-west-2.compute.internal: 136
ip-172-20-42-174.us-west-2.compute.internal: 45
ip-172-20-63-25.us-west-2.compute.internal: 271
ip-172-20-79-243.us-west-2.compute.internal: 136
ip-172-20-81-117.us-west-2.compute.internal: 37
ip-172-20-91-60.us-west-2.compute.internal: 281

after 5 min:

ip-172-20-100-20.us-west-2.compute.internal: 290
ip-172-20-101-115.us-west-2.compute.internal: 1
ip-172-20-35-34.us-west-2.compute.internal: 136
ip-172-20-42-174.us-west-2.compute.internal: 45
ip-172-20-63-25.us-west-2.compute.internal: 272
ip-172-20-79-243.us-west-2.compute.internal: 136
ip-172-20-81-117.us-west-2.compute.internal: 37
ip-172-20-91-60.us-west-2.compute.internal: 278

I deleted all of the weave pods again.

The process overall did NOT fix the stability issues on the original node.

Nor did the output from the /status/connections change on the original node:
admin@ip-172-20-81-117:~$ curl http://127.0.0.1:6784/status/connections
-> 172.20.79.243:6783 established fastdp 9a:74:7a:fd:4b:44(ip-172-20-79-243)
-> 172.20.101.115:6783 established fastdp ce:cf:f9:61:89:0b(ip-172-20-101-115)
-> 172.20.63.25:6783 established fastdp 76:fc:72:a9:29:9e(ip-172-20-63-25)
-> 172.20.35.34:6783 established fastdp 1a:cc:23:8a:b4:f1(ip-172-20-35-34)
-> 172.20.100.20:6783 established fastdp 7a:8c:e3:aa:36:c3(ip-172-20-100-20)
-> 172.20.91.60:6783 established fastdp ba:b4:a9:76:9a:e5(ip-172-20-91-60)
<- 172.20.42.174:46499 established fastdp 86:4e:8b:7e:79:59(ip-172-20-42-174)
-> 172.20.81.117:6783 failed cannot connect to ourself, retry: never

Maybe there were more steps to get weave to restart completely that I didn't know to take?

I'd love to hear that data from this experiment lead someone in some direction, even if just proves that it's not Socket depletion that's the culprit (but it sure feels like it to me).

bboreham · 2017-02-05T07:03:20Z

-> 172.20.81.117:6783 failed cannot connect to ourself, retry: never

Why can weave talk to itself???

Assuming you mean "why am I seeing this one line above", the reason is that "does this IP address connect back to myself" is a question that cannot in general be resolved without trying it (given NAT, public/private IPs, etc), so we don't bother to optimise out that case, we just try the address and blacklist it if it's us.

bboreham · 2017-02-05T07:09:01Z

Socket depletion

In the netstat logs I was sent, far and away the most sockets in use (thousands) were between kube-apiserver and kubelet on port 10250. I did not determine whether this is abnormal.

However, those dumps were from mostly-working nodes; it would be interesting to set up a netstat dump every five minutes on all nodes and then look at the last one before a failure.

bboreham · 2017-02-05T15:07:20Z

Or if you are scraping node metrics from /proc/net/sockstat (eg the Prometheus node exporter does this) that would also show if I'm on to something.

chrislovecnm · 2017-02-06T18:17:27Z

To update this.

The current problem we are diagnosing is that there is a delay making a connection from the host to the pod networks.

run curl -vvv http://10.41.0.0:8080/healthz-dnsmasq the health check for kube-dns takes > 20 secs
run curl -vvv http://10.41.0.0:8080/healthz-dnsmasq a second time and it responds immediately

The problem does not seem to be between pods, but between kubelet and the pods. Now I asked about the noncidr address space previously. Does that impact this routing?

How is the route / bridge configured from the host to the pod network? How can we compare a working system to this broken system??

bboreham · 2017-02-06T20:47:45Z

From a packet capture you supplied, I can see two interfaces have the same IP address - 10.37.0.0 - which is disrupting the conversations. Can you please run inside the weave-kube pod:

/usr/bin/weaveutil process-addrs weave

to list all the veths connected to the 'weave' bridge.

bboreham · 2017-02-06T20:47:54Z

How is the route / bridge configured from the host to the pod network?

It's a straightforward Linux bridge named weave. Each pod has a veth (virtual ethernet) device with one end presented as eth0 inside the pod's namespace and the other end connected to the bridge.

The bridge itself is then given an IP address in the same subnet as the pods, which creates an entry in the host's route table for that subnet (10.32.0.0/12).

chrislovecnm · 2017-02-07T03:01:03Z

To update the issue. For some reason multiple ip addresses existed on the weave bridge

admin@ip-172-20-81-117:~$ ip addr show dev weave
6: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc noqueue state UP group default qlen 1000
    link/ether 5a:a4:1d:20:42:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.43.128.0/12 scope global weave
       valid_lft forever preferred_lft forever
    inet 10.43.128.2/12 scope global secondary weave
       valid_lft forever preferred_lft forever
    inet6 fe80::58a4:1dff:fe20:42c0/64 scope link
       valid_lft forever preferred_lft forever

The work around is to remove the duplicate interfaces. We need to document more on the fix and mitigation.

chrislovecnm · 2017-02-07T16:45:45Z

@bboreham do you want a separate issue open for this bug?

bboreham · 2017-02-07T17:56:13Z

Once we identify specific bugs they should be separate; for now it's fine to keep all the comments in one place.

chrislovecnm · 2017-02-08T17:15:03Z

@bboreham any idea on how the duplicate ip was introduced, and what is a good mitigation process?

bboreham · 2017-02-08T18:08:15Z

Working on #2719 as a theory with #2610 as fix and #2612 to surface any similar problems.

Trying to reproduce the failure as a first step.

chrislovecnm · 2017-02-08T18:22:48Z

It has taken 12 or so hours to have the weave pods start to fail. So you may want to let a cluster run overnight. Also this seems to be only occuring on clusters that have some load on them.

bboreham · 2017-02-11T10:34:18Z

Chris supplied this event notification log:

[redacted]: pod weave-net-4ozht_kube-system(546acee0-ee25-11e6-8965-068f417b4097) evicted successfully

This led to #2794 which is my current favourite explanation for "restarting after a few hours": after a Demonset pod is evicted, it is immediately scheduled back to the same node.

rade · 2017-02-11T10:36:36Z

Interesting. Is the restarting actually that bad? In "plain" weave that would mostly go unnoticed by applications, but perhaps not in a k8s setting?

bboreham · 2017-02-11T10:42:46Z

I believe you mean that when fastdp is in use, already-connected pods would not experience any effect from Weave Net restarting.

If Kubernetes took some time to restart Weave Net (which is plausible if the machine is very loaded) then new pods could not connect and existing pods could not start new connections with remote pods.

However the eviction deletes Weave's persistence file which would otherwise avoid the race at #2784. Once you hit that race the node is unusable until rebooted.

rade · 2017-02-11T10:50:19Z

when fastdp is in use, already-connected pods would not experience any effect from Weave Net restarting.

Even without fastdp, most running apps will just see a blip in throughput.

If Kubernetes took some time to restart Weave Net

Right. That would be bad.

new pods could not connect and existing pods could not start new connections with remote pods.

I see. One would hope that with the right app architecture and health-checking etc policies this shouldn't cause significant disruption. But we have now kicked off a chain of events with unpredictable outcomes.

the eviction deletes Weave's persistence file

ouch

sstarcher · 2017-02-12T14:40:57Z

@bboreham during the original issue we were not seeing evictions at all we were seeing weave stopping itself due to the inability to connect to the api server.

The eviction issue is a new issue due to over allocating resources.

chrislovecnm · 2017-02-24T21:39:48Z

Closing this as it should be fixed in 1.9.1

alok87 · 2017-07-28T19:01:37Z

@chrislovecnm we are using weave 1.9.4 and we are facing the exact same issue with weave
and have the exact samee i/o timeout to api server in logs. is the fix for this released, is it part of 1.9.4?

rade · 2017-07-29T05:13:24Z

@alok87

we are using weave 1.9.4

May I ask why you are running that specific version? I am curious because we are seeing a lot of 1.9.4 out there and have been wondering for a while why that is the case.

As for the problem you are seeing, I believe the answer is in @bboreham's #2768 (comment).

alok87 · 2017-07-30T03:46:49Z

@rade we installed kubernetes 1.6.4 using kops and kops installed weave 1.9.4

We initially thought eviction could be the reason as our logs were getting spammed. But the issue was happening even after we controlled log spamming(we faced only disk evictions). Issue has reduced after we have docker rm the Dead containers (every few hours). But we are still not sure of the root cause. Few questions

Root cause: Could this issue be related to kube api server performance issue as it timedout. I see spikes to 12MB/s from 2MB/s in the Kube API Server Network monitoring in grafana at the same time.
Is the issue discussed above merged? if yes could you suggest us the version.
We route our traffic from AWS ELB to the nodes at node port. Weave-net runs as daemon set. Whenever weave-net restarts we get backend connection errors - which is not right as weave-net restart should not impact ongoing traffic? Correct? Should I open a separate issue for this?

rade · 2017-07-31T05:29:37Z

we installed kubernetes 1.6.4 using kops and kops installed weave 1.9.4

That's odd. kops should be installing Weave Net 1.9.8. as of kops 1.6.2. Are you running an older version?

Is the issue discussed above merged? if yes could you suggest us the version.

The issue @bboreham mentioned is #2794, which is resolved Weave Net 1.9.6. I suggest you use the latest 1.6.x kops, which, as per the above, should install Weave Net 1.9.8.

However, note that the fix actually isn't in Weave Net itself but only the documentation, namely we added a section on pod eviction. So I suggest you check that the yamls for the Weave Net Daemonset deployed in your cluster do actually set resource limits as described in the docs.

Whenever weave-net restarts we get backend connection errors - which is not right as weave-net restart should not impact ongoing traffic? Correct?

Sounds like #3073.

marccarre self-assigned this Feb 3, 2017

marccarre assigned chrislovecnm and unassigned marccarre Feb 3, 2017

marccarre added the state/investigating label Feb 3, 2017

marccarre removed the state/investigating label Feb 3, 2017

marccarre unassigned chrislovecnm Feb 3, 2017

chrislovecnm mentioned this issue Feb 7, 2017

Weave issue with duplicate ip addresses kubernetes/kops#1819

Closed

bboreham mentioned this issue Feb 9, 2017

Race in reclaiming existing addresses #2784

Closed

bboreham added state/investigating [component/kube] labels Feb 9, 2017

chrislovecnm closed this as completed Feb 24, 2017

bboreham removed the state/investigating label May 30, 2017

bboreham added this to the n/a milestone May 30, 2017

alok87 mentioned this issue Jul 29, 2017

kubelet is spamming "daemon.log" with error messages kubernetes/kubernetes#47294

Closed

Weave restarting after a few hours #2768

Weave restarting after a few hours #2768

Comments

chrislovecnm commented Feb 3, 2017 • edited Loading

chrislovecnm commented Feb 3, 2017 • edited Loading

chrislovecnm commented Feb 3, 2017

chrislovecnm commented Feb 3, 2017

chrislovecnm commented Feb 3, 2017

marccarre commented Feb 3, 2017

chrislovecnm commented Feb 3, 2017 • edited Loading

chrislovecnm commented Feb 3, 2017

chrislovecnm commented Feb 3, 2017

bboreham commented Feb 3, 2017

chrislovecnm commented Feb 4, 2017

bboreham commented Feb 4, 2017

chrislovecnm commented Feb 4, 2017

chrislovecnm commented Feb 4, 2017

chrislovecnm commented Feb 4, 2017

chrislovecnm commented Feb 4, 2017

bboreham commented Feb 4, 2017

jroberts235 commented Feb 5, 2017 • edited Loading

bboreham commented Feb 5, 2017

bboreham commented Feb 5, 2017 • edited Loading

bboreham commented Feb 5, 2017

chrislovecnm commented Feb 6, 2017

bboreham commented Feb 6, 2017 • edited Loading

bboreham commented Feb 6, 2017

chrislovecnm commented Feb 7, 2017

chrislovecnm commented Feb 7, 2017

bboreham commented Feb 7, 2017

chrislovecnm commented Feb 8, 2017

bboreham commented Feb 8, 2017

chrislovecnm commented Feb 8, 2017

bboreham commented Feb 11, 2017

rade commented Feb 11, 2017

bboreham commented Feb 11, 2017

rade commented Feb 11, 2017

sstarcher commented Feb 12, 2017

chrislovecnm commented Feb 24, 2017

alok87 commented Jul 28, 2017

rade commented Jul 29, 2017

alok87 commented Jul 30, 2017 • edited Loading

rade commented Jul 31, 2017

chrislovecnm commented Feb 3, 2017 •

edited

Loading

chrislovecnm commented Feb 3, 2017 •

edited

Loading

chrislovecnm commented Feb 3, 2017 •

edited

Loading

jroberts235 commented Feb 5, 2017 •

edited

Loading

bboreham commented Feb 5, 2017 •

edited

Loading

bboreham commented Feb 6, 2017 •

edited

Loading

alok87 commented Jul 30, 2017 •

edited

Loading