Skip to content
This repository has been archived by the owner on Sep 7, 2023. It is now read-only.

[ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8302->10.254.0.7:8302: sendto: operation not permitted #60

Closed
prologic opened this issue Jun 7, 2017 · 5 comments
Closed

[ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8302->10.254.0.7:8302: sendto: operation not permitted #60

prologic opened this issue Jun 7, 2017 · 5 comments

Comments

@prologic
Copy link

prologic commented Jun 7, 2017

HI,

I'm getting some errors in the seed and the nodes that I don't yet fully understand.

I'm using this docker stack / docker-compose.yml and spinning up the service(s) with:

$ docker stack deploy -c consul.yml consul

Seed logs:

==> Starting Consul agent...
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Consul agent running!
           Version: 'v0.8.3'
           Node ID: 'af95ca51-bce1-b19b-e3c5-b8bf57df02f1'
         Node name: 'seed'
        Datacenter: 'dc1'
            Server: true (bootstrap: false)
       Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 10.254.0.7 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas: <disabled>

==> Log data will now stream in as it occurs:

    2017/06/07 06:41:28 [INFO] raft: Initial configuration (index=0): []
    2017/06/07 06:41:28 [INFO] raft: Node at 10.254.0.7:8300 [Follower] entering Follower state (Leader: "")
    2017/06/07 06:41:28 [INFO] serf: EventMemberJoin: seed 10.254.0.7
    2017/06/07 06:41:28 [INFO] serf: EventMemberJoin: seed.dc1 10.254.0.7
    2017/06/07 06:41:28 [INFO] consul: Adding LAN server seed (Addr: tcp/10.254.0.7:8300) (DC: dc1)
    2017/06/07 06:41:28 [INFO] consul: Handled member-join event for server "seed.dc1" in area "wan"
    2017/06/07 06:41:35 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:41:36 [WARN] raft: no known peers, aborting election
    2017/06/07 06:41:57 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:41:57 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:41:57 [INFO] serf: EventMemberJoin: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:41:57 [INFO] consul: Handled member-join event for server "6d158c5b5562.dc1" in area "wan"
    2017/06/07 06:41:57 [INFO] serf: EventMemberJoin: f88dacfafd73 10.255.0.16
    2017/06/07 06:41:57 [INFO] consul: Adding LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:41:58 [INFO] serf: EventMemberJoin: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:41:59 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:42:02 [ERR] agent: coordinate update error: No cluster leader
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=b7827419-5378-9b71-0bcc-67ea0039da7d&version=0.8.3: dial tcp: lookup checkpoint-api.hashicorp.com on 127.0.0.11:53: read udp 127.0.0.1:58949->127.0.0.11:53: i/o timeout
    2017/06/07 06:42:07 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 1s...
    2017/06/07 06:42:09 [INFO] memberlist: Suspect 1ae40b55c20b has failed, no acks received
    2017/06/07 06:42:09 [ERR] memberlist: Push/Pull with f88dacfafd73 failed: dial tcp 10.255.0.16:8301: i/o timeout
    2017/06/07 06:42:14 [INFO] memberlist: Marking 1ae40b55c20b as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:42:14 [INFO] serf: EventMemberFailed: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:42:18 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 2s...
    2017/06/07 06:42:19 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:42:24 [INFO] memberlist: Marking f88dacfafd73 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:42:24 [INFO] serf: EventMemberFailed: f88dacfafd73 10.255.0.16
    2017/06/07 06:42:27 [INFO] serf: EventMemberJoin: f88dacfafd73.dc1 10.255.0.16
    2017/06/07 06:42:27 [INFO] consul: Handled member-join event for server "f88dacfafd73.dc1" in area "wan"
    2017/06/07 06:42:28 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:42:28 [INFO] serf: attempting reconnect to f88dacfafd73 10.255.0.16:8301
    2017/06/07 06:42:30 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:42:30 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 4s...
    2017/06/07 06:42:32 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:42:35 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:42:35 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:42:36 [INFO] memberlist: Suspect 6d158c5b5562.dc1 has failed, no acks received
    2017/06/07 06:42:41 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:42:44 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 8s...
    2017/06/07 06:42:56 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:42:57 [INFO] serf: EventMemberJoin: f88dacfafd73 10.255.0.16
    2017/06/07 06:42:58 [INFO] serf: EventMemberJoin: 1ae40b55c20b.dc1 10.255.0.10
    2017/06/07 06:42:58 [INFO] consul: Handled member-join event for server "1ae40b55c20b.dc1" in area "wan"
    2017/06/07 06:43:01 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:43:02 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 16s...
    2017/06/07 06:43:03 [INFO] serf: EventMemberJoin: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:43:06 [INFO] memberlist: Marking 6d158c5b5562.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:06 [INFO] serf: EventMemberFailed: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:43:06 [INFO] consul: Handled member-failed event for server "6d158c5b5562.dc1" in area "wan"
    2017/06/07 06:43:09 [INFO] memberlist: Suspect 6d158c5b5562.dc1 has failed, no acks received
    2017/06/07 06:43:10 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:43:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:43:15 [INFO] memberlist: Marking f88dacfafd73 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:15 [INFO] serf: EventMemberFailed: f88dacfafd73 10.255.0.16
    2017/06/07 06:43:19 [ERR] memberlist: Push/Pull with f88dacfafd73 failed: dial tcp 10.255.0.16:8301: i/o timeout
    2017/06/07 06:43:20 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:43:21 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:43:26 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:43:28 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 32s...
    2017/06/07 06:43:31 [INFO] memberlist: Suspect 1ae40b55c20b has failed, no acks received
    2017/06/07 06:43:32 [ERR] memberlist: Push/Pull with 1ae40b55c20b.dc1 failed: dial tcp 10.255.0.10:8302: i/o timeout
    2017/06/07 06:43:36 [INFO] memberlist: Marking 1ae40b55c20b as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:36 [INFO] serf: EventMemberFailed: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:43:38 [INFO] serf: attempting reconnect to 1ae40b55c20b 10.255.0.10:8301
    2017/06/07 06:43:41 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:43:42 [INFO] memberlist: Suspect f88dacfafd73.dc1 has failed, no acks received
    2017/06/07 06:43:46 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:46 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:43:47 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:43:53 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:44:02 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:44:08 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: rpc error: lead thread didn't get connection. Retrying in 1s...
    2017/06/07 06:44:12 [INFO] memberlist: Marking f88dacfafd73.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:44:12 [INFO] serf: EventMemberFailed: f88dacfafd73.dc1 10.255.0.16
    2017/06/07 06:44:12 [INFO] consul: Handled member-failed event for server "f88dacfafd73.dc1" in area "wan"
    2017/06/07 06:44:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:44:15 [INFO] memberlist: Suspect f88dacfafd73.dc1 has failed, no acks received
    2017/06/07 06:44:18 [INFO] serf: attempting reconnect to f88dacfafd73 10.255.0.16:8301
    2017/06/07 06:44:19 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 2s...
    2017/06/07 06:44:22 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:44:24 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:44:28 [INFO] serf: attempting reconnect to 6d158c5b5562.dc1 10.255.0.13:8302
    2017/06/07 06:44:29 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:44:29 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:44:29 [ERR] memberlist: Push/Pull with 6d158c5b5562 failed: dial tcp 10.255.0.13:8301: i/o timeout
    2017/06/07 06:44:31 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 4s...
    2017/06/07 06:44:34 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:44:37 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:44:45 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 8s...
    2017/06/07 06:44:48 [INFO] memberlist: Suspect 1ae40b55c20b.dc1 has failed, no acks received
    2017/06/07 06:44:51 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:44:53 [INFO] serf: EventMemberJoin: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:44:53 [INFO] serf: EventMemberJoin: f88dacfafd73 10.255.0.16
    2017/06/07 06:44:58 [INFO] serf: attempting reconnect to 6d158c5b5562 10.255.0.13:8301
    2017/06/07 06:45:01 [INFO] serf: EventMemberJoin: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:45:01 [INFO] consul: Handled member-join event for server "6d158c5b5562.dc1" in area "wan"
    2017/06/07 06:45:02 [ERR] memberlist: Push/Pull with 1ae40b55c20b.dc1 failed: dial tcp 10.255.0.10:8302: i/o timeout
    2017/06/07 06:45:03 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 16s...
    2017/06/07 06:45:04 [INFO] memberlist: Suspect 1ae40b55c20b has failed, no acks received
    2017/06/07 06:45:09 [INFO] memberlist: Marking 1ae40b55c20b as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:09 [INFO] serf: EventMemberFailed: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:45:09 [ERR] memberlist: Push/Pull with 1ae40b55c20b failed: dial tcp 10.255.0.10:8301: i/o timeout
    2017/06/07 06:45:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:45:13 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:45:14 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:45:18 [INFO] memberlist: Marking 1ae40b55c20b.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:18 [INFO] serf: EventMemberFailed: 1ae40b55c20b.dc1 10.255.0.10
    2017/06/07 06:45:18 [INFO] consul: Handled member-failed event for server "1ae40b55c20b.dc1" in area "wan"
    2017/06/07 06:45:19 [INFO] memberlist: Marking f88dacfafd73 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:19 [INFO] serf: EventMemberFailed: f88dacfafd73 10.255.0.16
    2017/06/07 06:45:21 [INFO] memberlist: Suspect 1ae40b55c20b.dc1 has failed, no acks received
    2017/06/07 06:45:25 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:45:27 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:45:28 [INFO] serf: attempting reconnect to f88dacfafd73.dc1 10.255.0.16:8302
    2017/06/07 06:45:29 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 32s...
    2017/06/07 06:45:30 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:30 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:45:36 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:45:37 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:45:38 [INFO] serf: attempting reconnect to 6d158c5b5562 10.255.0.13:8301
    2017/06/07 06:45:56 [INFO] memberlist: Suspect 6d158c5b5562.dc1 has failed, no acks received
    2017/06/07 06:46:01 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:46:01 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.254.0.7:8300,10.255.0.13:8300,10.255.0.16:8300
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:04 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2017/06/07 06:46:04 [INFO] raft: Node at 10.254.0.7:8300 [Candidate] entering Candidate state in term 2
    2017/06/07 06:46:10 [WARN] raft: Election timeout reached, restarting election
    2017/06/07 06:46:10 [INFO] raft: Node at 10.254.0.7:8300 [Candidate] entering Candidate state in term 3
    2017/06/07 06:46:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:46:13 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.16:8300 10.255.0.16:8300}: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout
    2017/06/07 06:46:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.13:8300 10.255.0.13:8300}: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout
    2017/06/07 06:46:14 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:46:18 [INFO] serf: attempting reconnect to f88dacfafd73 10.255.0.16:8301
    2017/06/07 06:46:19 [WARN] raft: Election timeout reached, restarting election
    2017/06/07 06:46:19 [INFO] raft: Node at 10.254.0.7:8300 [Candidate] entering Candidate state in term 4
    2017/06/07 06:46:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.16:8300 10.255.0.16:8300}: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout
    2017/06/07 06:46:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.13:8300 10.255.0.13:8300}: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout
    2017/06/07 06:46:25 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:46:26 [INFO] memberlist: Marking 6d158c5b5562.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:46:26 [INFO] serf: EventMemberFailed: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:46:26 [INFO] consul: Handled member-failed event for server "6d158c5b5562.dc1" in area "wan"
==> Caught signal: terminated
    2017/06/07 06:46:28 [INFO] agent: requesting shutdown
    2017/06/07 06:46:28 [INFO] consul: shutting down server
    2017/06/07 06:46:28 [WARN] serf: Shutdown without a Leave
    2017/06/07 06:46:28 [ERR] dns: error starting tcp server: accept tcp 127.0.0.1:8600: use of closed network connection
    2017/06/07 06:46:28 [WARN] serf: Shutdown without a Leave
    2017/06/07 06:46:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.16:8300 10.255.0.16:8300}: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout
    2017/06/07 06:46:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.13:8300 10.255.0.13:8300}: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout
    2017/06/07 06:46:29 [INFO] agent: shutdown complete

Sample node logs:

    2017/06/07 06:54:49 [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8301->10.254.0.7:8301: sendto: operation not permitted
    2017/06/07 06:54:49 [ERR] memberlist: Push/Pull with seed.dc1 failed: dial tcp 10.254.0.7:8302: getsockopt: no route to host
    2017/06/07 06:54:51 [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8301->10.255.0.13:8301: sendto: operation not permitted
    2017/06/07 06:54:52 [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8301->10.254.0.7:8301: sendto: operation not permitted
    2017/06/07 06:54:52 [ERR] memberlist: Push/Pull with seed failed: dial tcp 10.254.0.7:8301: getsockopt: no route to host
@prologic prologic changed the title Operation Not Permitted [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8302->10.254.0.7:8302: sendto: operation not permitted Jun 7, 2017
@lnshi
Copy link

lnshi commented Jun 16, 2017

@prologic

The problem is related to three parts:

  1. ur -bind={{ GetInterfaceIP "eth0" }}
  2. ur ports port mapping
  3. how docker routing mesh works

1 make all consul traffic will go through physical IP eth0 and the corresponding port
2 u just map the container port to physical host also
with 3, here

You can rely on the routing mesh. When you publish a service port, the swarm makes the service accessible at the target port on every node, regardless of whether there is a task for the service running on that node or not. This is less complex and is the right choice for many types of services.

u can see how many services are trying to use one port in ur each physical node, so : port conflicting.

solution, map the port only to the host which the service is running, which is the host mode

ports:
  - target: 65301
    published: 65301
    mode: host

don't forget use above syntax with docker-compose version 3.2

@slackpad
Copy link
Contributor

Thanks @lnshi for fielding this one! @prologic can we close this out?

@prologic
Copy link
Author

prologic commented Jun 24, 2017 via email

@prologic
Copy link
Author

@lnshi I just had a chance to review your comments and I'm a bit confused to say the least. What's the point of using an overlay network if I'm just going to bind the conteiner's ports to the host? What am I missing here? The whole point of my (half baked) consul.yml spec above is to spin up a consul cluster as a docker service for things that need a distributed kvstore.

@prologic
Copy link
Author

prologic commented Jun 26, 2017 •
edited
Loading

Finally got this working:

consul.yml:

version: '3.2'

#customise this with options from
#https://www.consul.io/docs/agent/options.html

services:
  seed:
    hostname: seed
    image: consul:latest
    deploy:
      restart_policy:
        condition: none  #we do not want this to be restarted on timeout (see entrypoint options below)
      replicas: 1
      placement:
        constraints:
          - "node.labels.access == temp"
          - "node.labels.access != consul"
    environment:
      - "CONSUL_LOCAL_CONFIG={\"disable_update_check\": true}"
      - "CONSUL_BIND_INTERFACE=eth0"
    entrypoint:
      - timeout     #this seed fires up the cluster after which it is no longer needed
      - -sTERM      #this is the same signal as docker would send on a scale down / stop
      - -t300       #terminate after 5 mins
      - consul
      - agent
      - -server
      - -bootstrap-expect=3
      - -data-dir=/tmp/consuldata
      - -bind={{ GetInterfaceIP "eth0" }}
    networks:
      - "consul"

  cluster:
    image: consul:latest
    depends_on:
      - "seed"
    deploy:
      mode: global                                      ##this will deploy to all nodes that
      placement:
        constraints:
          - "node.labels.access == consul"            ##have the consul label
          - "node.labels.access != temp"
    environment:
      - "CONSUL_LOCAL_CONFIG={\"disable_update_check\": true}"
      - "CONSUL_BIND_INTERFACE=eth0"
      - "CONSUL_HTTP_ADDR=0.0.0.0"
    entrypoint:
      - consul
      - agent
      - -server
      - -data-dir=/tmp/consuldata
      - -bind={{ GetInterfaceIP "eth0" }}
      - -client=0.0.0.0
      - -retry-join=seed:8301
      - -ui                                              ##assuming you want the UI on
    networks:
      - "consul"
    ports:
      - "8500:8500"
      - "8600:8600"

networks:
  consul:
    driver: overlay

Usage:

$ docker stack deploy -c consul.yml consul
$ docker stack ls
NAME    SERVICES
consul  2

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                   PORTS
7x6s8vxl8csc        consul_cluster      global              3/3                 consul:latest           *:8500->8500/tcp,*:8600->8600/tcp
cl4f9ob98oji        consul_seed         replicated          1/1                 consul:latest

@realzero0 realzero0 mentioned this issue Jul 2, 2024
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@slackpad @prologic @lnshi