Running the official docker image for etcd will listen only on localhost. #8230

ahoma-paymentus · 2017-07-07T14:08:18Z

Here is the sample docker-compose.yml that demonstrates the problem.

version: '2'

services:
  etcd:
    image: quay.io/coreos/etcd
    # asking etcd to listen on all interfaces fails to register
    #command: sh -c 'etcd -listen-client-urls http://0.0.0.0:2379 -advertise-client-urls http://etcd-srv:2379'
    
    # Default etcd command only listens on localhost
    command: sh -c 'ifconfig eth0 | grep inet; etcd'

  etcd-check:
    image: quay.io/coreos/etcd
    depends_on:
      - etcd
    command: sh -c 'for f in 1 2 3;do sleep 5; echo "Check $$f $$(date +%H:%M:%S)"; etcdctl --endpoints http://etcd:2379 set /message Hello; sleep 3; done'

Here is the log:

$ docker-compose down --remove-orphans; docker-compose up -d; docker-compose logs -f
Stopping etcdtest_etcd_1 ...
Stopping etcdtest_etcd_1 ... done
Removing etcdtest_etcd-check_1 ... done
Removing etcdtest_etcd_1 ... done
Removing network etcdtest_default
Creating network "etcdtest_default" with the default driver
Creating etcdtest_etcd_1 ...
Creating etcdtest_etcd_1 ... done
Creating etcdtest_etcd-check_1 ...
Creating etcdtest_etcd-check_1 ... done
Attaching to etcdtest_etcd-check_1, etcdtest_etcd_1
etcd_1        |           inet addr:172.22.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
etcd_1        | 2017-07-07 13:59:06.820064 I | etcdmain: etcd Version: 3.2.1
etcd_1        | 2017-07-07 13:59:06.820183 I | etcdmain: Git SHA: 61fc123
etcd_1        | 2017-07-07 13:59:06.820384 I | etcdmain: Go Version: go1.8.3
etcd_1        | 2017-07-07 13:59:06.820400 I | etcdmain: Go OS/Arch: linux/amd64
etcd_1        | 2017-07-07 13:59:06.820425 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
etcd_1        | 2017-07-07 13:59:06.820435 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
etcd_1        | 2017-07-07 13:59:06.820656 I | embed: listening for peers on http://localhost:2380
etcd_1        | 2017-07-07 13:59:06.820774 I | embed: listening for client requests on localhost:2379
etcd_1        | 2017-07-07 13:59:06.824427 I | etcdserver: name = default
etcd_1        | 2017-07-07 13:59:06.824465 I | etcdserver: data dir = default.etcd
etcd_1        | 2017-07-07 13:59:06.824474 I | etcdserver: member dir = default.etcd/member
etcd_1        | 2017-07-07 13:59:06.824480 I | etcdserver: heartbeat = 100ms
etcd_1        | 2017-07-07 13:59:06.824486 I | etcdserver: election = 1000ms
etcd_1        | 2017-07-07 13:59:06.824497 I | etcdserver: snapshot count = 100000
etcd_1        | 2017-07-07 13:59:06.824511 I | etcdserver: advertise client URLs = http://localhost:2379
etcd_1        | 2017-07-07 13:59:06.824523 I | etcdserver: initial advertise peer URLs = http://localhost:2380
etcd_1        | 2017-07-07 13:59:06.824533 I | etcdserver: initial cluster = default=http://localhost:2380
etcd_1        | 2017-07-07 13:59:06.829802 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
etcd_1        | 2017-07-07 13:59:06.829838 I | raft: 8e9e05c52164694d became follower at term 0
etcd_1        | 2017-07-07 13:59:06.829852 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
etcd_1        | 2017-07-07 13:59:06.829860 I | raft: 8e9e05c52164694d became follower at term 1
etcd_1        | 2017-07-07 13:59:06.930646 W | auth: simple token is not cryptographically signed
etcd_1        | 2017-07-07 13:59:06.936427 I | etcdserver: starting server... [version: 3.2.1, cluster version: to_be_decided]
etcd_1        | 2017-07-07 13:59:06.937864 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
etcd_1        | 2017-07-07 13:59:07.630189 I | raft: 8e9e05c52164694d is starting a new election at term 1
etcd_1        | 2017-07-07 13:59:07.630272 I | raft: 8e9e05c52164694d became candidate at term 2
etcd_1        | 2017-07-07 13:59:07.630298 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
etcd_1        | 2017-07-07 13:59:07.630314 I | raft: 8e9e05c52164694d became leader at term 2
etcd_1        | 2017-07-07 13:59:07.630337 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
etcd_1        | 2017-07-07 13:59:07.630574 I | etcdserver: setting up the initial cluster version to 3.2
etcd_1        | 2017-07-07 13:59:07.632363 N | etcdserver/membership: set the initial cluster version to 3.2
etcd_1        | 2017-07-07 13:59:07.632442 I | etcdserver/api: enabled capabilities for version 3.2
etcd_1        | 2017-07-07 13:59:07.632544 I | embed: ready to serve client requests
etcd_1        | 2017-07-07 13:59:07.632928 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
etcd_1        | 2017-07-07 13:59:07.632968 I | etcdserver: published {Name:default ClientURLs:[http://localhost:2379]} to cluster cdf818194e3a8c32
etcd-check_1  | Check 1 13:59:12
etcd-check_1  | Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.22.0.2:2379: getsockopt: connection refused
etcd-check_1  |
etcd-check_1  | error #0: dial tcp 172.22.0.2:2379: getsockopt: connection refused
etcd-check_1  |
etcd-check_1  | Check 2 13:59:20
etcd-check_1  | Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.22.0.2:2379: getsockopt: connection refused
etcd-check_1  |
etcd-check_1  | error #0: dial tcp 172.22.0.2:2379: getsockopt: connection refused
etcd-check_1  |
etcd-check_1  | Check 3} 13:59:28
etcd-check_1  | Error:  client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 172.22.0.2:2379: getsockopt: connection refused
etcd-check_1  |
etcd-check_1  | error #0: dial tcp 172.22.0.2:2379: getsockopt: connection refused
etcd-check_1  |
etcdtest_etcd-check_1 exited with code 0

The text was updated successfully, but these errors were encountered:

gyuho · 2017-07-07T14:12:49Z

getsockopt: connection refused

Seems like docker configuration issue, not etcd issue.

ahoma-paymentus · 2017-07-07T14:19:15Z

It is not. Ping works fine. However curl does not.

gyuho · 2017-07-07T15:40:51Z

It is not. Ping works fine. However curl does not.

Shouldn't matter. Your etcd cluster started fine. Seems like you just need to expose port 2379.

ahoma-paymentus · 2017-07-07T16:33:10Z

The logs shows that etcd is listening on localhost only.
Because of that etcdctl works on

gyuho · 2017-07-07T16:35:24Z

The logs shows that etcd is listening on localhost only.

Yeah you need to either use host network for your etcd container, or expose client port 2379 to the container that your etcdctl is running.

ahoma-paymentus · 2017-07-07T21:25:28Z

Here is a more detailed run. This clearly shows that etcd is not registering on anything but 127.0.0.1

> cat docker-compose.yml
version: '2'
services:
  etcd:
    image: quay.io/coreos/etcd
    ports:
      - "2379:2379"
      - "2380:2380"
    command: sh -c 'apk update; apk add curl; ifconfig eth0 | grep inet; etcd'
  etcd-check:
    image: quay.io/coreos/etcd
    depends_on:
      - etcd
    command: sh -c 'apk update; apk add curl ;while true;do sleep 1;done'

> docker-compose up -d
...
> docker-compose ps
        Name                       Command               State                       Ports
---------------------------------------------------------------------------------------------------------------
etcdtest_etcd-check_1   sh -c apk update; apk add  ...   Up      2379/tcp, 2380/tcp
etcdtest_etcd_1         sh -c apk update; apk add  ...   Up      0.0.0.0:2379->2379/tcp, 0.0.0.0:2380->2380/tcp

# run curl inside of container, localhost ip works
> docker-compose exec etcd curl -sS http://127.0.0.1:2379/version
{"etcdserver":"3.2.2","etcdcluster":"3.2.0"}

# extract IP of container
> container_ip=$(docker-compose exec etcd ifconfig eth0|grep -oP 'inet addr:\K\S+');echo container_ip=${container_ip}
container_ip=172.19.0.2

# run curl outside, using exposed port, it does not work
> docker-compose exec etcd curl -sS http://${container_ip}:2379/version
curl: (7) Failed to connect to 172.19.0.2 port 2379: Connection refused

# run curl on 
> curl -sS http://localhost:2379/version
curl: (56) Recv failure: Connection reset by peer

run on AWS instance

heyitsanthony · 2017-07-07T21:40:03Z

@ahoma-paymentus launching etcd as etcd with no arguments will only bind to localhost by design to avoid unintentionally exposing the etcd instance to the outside world.

The commented out line:

#command: sh -c 'etcd -listen-client-urls http://0.0.0.0:2379 -advertise-client-urls http://etcd-srv:2379'

is closer to what should be passed to etcd in order to expose it to the rest of the network via docker.

ahoma-paymentus · 2017-07-09T18:23:39Z

And this is exactly the problem. etcd refuses to listen on all ports when specifying those options:

ahoma-paymentus · 2017-07-09T18:24:22Z

Running:
command: sh -c 'etcd -listen-client-urls http://0.0.0.0:2379 -advertise-client-urls http://etcd:2379,http://127.0.0.1:2379'

etcd fails to listen

etcd_1        | 2017-07-09 18:22:40.586119 I | etcdmain: etcd Version: 3.2.1
etcd_1        | 2017-07-09 18:22:40.586529 I | etcdmain: Git SHA: 61fc123
etcd_1        | 2017-07-09 18:22:40.586586 I | etcdmain: Go Version: go1.8.3
etcd_1        | 2017-07-09 18:22:40.586622 I | etcdmain: Go OS/Arch: linux/amd64
etcd_1        | 2017-07-09 18:22:40.586751 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
etcd_1        | 2017-07-09 18:22:40.586815 W | etcdmain: no data-dir provided, using default data-dir ./default.etcd
etcd_1        | 2017-07-09 18:22:40.587605 I | embed: listening for peers on http://localhost:2380
etcd_1        | 2017-07-09 18:22:40.587769 I | embed: listening for client requests on 0.0.0.0:2379
etcd_1        | 2017-07-09 18:22:40.593339 I | etcdserver: name = default
etcd_1        | 2017-07-09 18:22:40.593447 I | etcdserver: data dir = default.etcd
etcd_1        | 2017-07-09 18:22:40.593462 I | etcdserver: member dir = default.etcd/member
etcd_1        | 2017-07-09 18:22:40.593526 I | etcdserver: heartbeat = 100ms
etcd_1        | 2017-07-09 18:22:40.593624 I | etcdserver: election = 1000ms
etcd_1        | 2017-07-09 18:22:40.593658 I | etcdserver: snapshot count = 100000
etcd_1        | 2017-07-09 18:22:40.593703 I | etcdserver: advertise client URLs = http://127.0.0.1:2379,http://etcd:2379
etcd_1        | 2017-07-09 18:22:40.593752 I | etcdserver: initial advertise peer URLs = http://localhost:2380
etcd_1        | 2017-07-09 18:22:40.593818 I | etcdserver: initial cluster = default=http://localhost:2380
etcd_1        | 2017-07-09 18:22:40.691991 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
etcd_1        | 2017-07-09 18:22:40.692195 I | raft: 8e9e05c52164694d became follower at term 0
etcd_1        | 2017-07-09 18:22:40.692421 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
etcd_1        | 2017-07-09 18:22:40.692438 I | raft: 8e9e05c52164694d became follower at term 1
etcd_1        | 2017-07-09 18:22:40.708973 W | auth: simple token is not cryptographically signed
etcd_1        | 2017-07-09 18:22:40.715475 I | etcdserver: starting server... [version: 3.2.1, cluster version: to_be_decided]
etcd_1        | 2017-07-09 18:22:40.716808 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
etcd_1        | 2017-07-09 18:22:41.293152 I | raft: 8e9e05c52164694d is starting a new election at term 1
etcd_1        | 2017-07-09 18:22:41.293755 I | raft: 8e9e05c52164694d became candidate at term 2
etcd_1        | 2017-07-09 18:22:41.294442 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
etcd_1        | 2017-07-09 18:22:41.294604 I | raft: 8e9e05c52164694d became leader at term 2
etcd_1        | 2017-07-09 18:22:41.294970 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
etcd_1        | 2017-07-09 18:22:41.295817 I | etcdserver: published {Name:default ClientURLs:[http://127.0.0.1:2379 http://etcd:2379]} to cluster cdf818194e3a8c32
etcd_1        | 2017-07-09 18:22:41.296049 I | etcdserver: setting up the initial cluster version to 3.2
etcd_1        | 2017-07-09 18:22:41.296403 I | embed: ready to serve client requests
etcd_1        | 2017-07-09 18:22:41.297377 N | embed: serving insecure client requests on [::]:2379, this is strongly discouraged!
etcd_1        | 2017-07-09 18:22:41.297738 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {[::]:2379 <nil>}
etcd_1        | 2017-07-09 18:22:41.297814 I | etcdserver/api/v3rpc: grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp [::]:2379: connect: cannot assign requested address"; Reconnecting to {[::]:2379 <nil>}

gyuho · 2017-07-09T18:39:45Z

Try https://docs.docker.com/compose/compose-file/compose-file-v2/#expose

ahoma-paymentus · 2017-07-09T19:01:38Z

Again the expose command has nothing to do with... moving on..

I run the following command:
command: sh -c 'local_ip=$$(ifconfig eth0|sed -n -e "s/.inet addr:$[0-9.][0-9.]$.*/\1/p");echo $$local_ip; etcd -listen-client-urls http://$$local_ip:2379,http://127.0.0.1:2379 -advertise-client-urls http://$$local_ip:2379,http://127.0.0.1:2379'

And this worked. So it looks etcd is not working when using -listen-client-urls http://0.0.0.0:2379, however it works when it is provided with local IP explicitly.

Sound like a bug.

hexfusion · 2017-07-09T23:08:50Z

@ahoma-paymentus FWIW I can not replicate your issue locally.

docker-compose.yml

version: '2'
services:
  etcd:
    image: quay.io/coreos/etcd
    ports:
      - "2379:2379"
      - "2380:2380"
    command: sh -c 'apk update; apk add curl;etcd -listen-client-urls http://0.0.0.0:2379 -advertise-client-urls http://etcd-srv:2379'
  etcd-check:
    image: quay.io/coreos/etcd
    depends_on:
      - etcd
    command: sh -c 'apk update; apk add curl;while true;do sleep 1;done'

$ docker-compose up -d
Creating docker_etcd_1
Creating docker_etcd-check_1

$ docker-compose exec etcd curl -sS http://127.0.0.1:2379/version
{"etcdserver":"3.2.2","etcdcluster":"3.2.0”}

$ container_ip=$(docker-compose exec etcd ifconfig eth0|grep -oP 'inet addr:\K\S+');echo container_ip=${container_ip}
container_ip=172.18.0.2

$ docker-compose exec etcd curl -sS http://172.18.0.2:2379/version
{"etcdserver":"3.2.2","etcdcluster":"3.2.0”}

$ curl -sS http://localhost:2379/version
{"etcdserver":"3.2.2","etcdcluster":"3.2.0"}

ahoma-paymentus · 2017-07-10T15:17:32Z

I think I found the problem. When running it on Docker Windows 10, it looks the containers are not running using IP6. Running ifconfig in the containers shows only IP4 address.

etcd is trying to register on IP6 and it fails.

Running the same on AWS image ifconfig returns IP4 and IP6 address. And etcd works.

This problem is sidestepped when asking etcd to register on a specific IP4 ip.

It looks etcd thinks the container is using IP6 when it's not.
Maybe a command line option to limit to IP4 would be useful to fix this problem?

heyitsanthony · 2017-07-12T04:49:25Z

@ahoma-paymentus that should be fixed by #8221; it'll be in 3.2.3.

gyuho closed this as completed Jul 7, 2017

ihcsim mentioned this issue Feb 12, 2018

etcd errors running swarm local example ihcsim/stolon-swarm#5

Closed

lewisheadden mentioned this issue Oct 13, 2018

Migrate etcd features from Spring Cloud and update etcd client spring-projects/spring-integration-extensions#198

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running the official docker image for etcd will listen only on localhost. #8230

Running the official docker image for etcd will listen only on localhost. #8230

ahoma-paymentus commented Jul 7, 2017

gyuho commented Jul 7, 2017

ahoma-paymentus commented Jul 7, 2017

gyuho commented Jul 7, 2017

ahoma-paymentus commented Jul 7, 2017

gyuho commented Jul 7, 2017

ahoma-paymentus commented Jul 7, 2017

heyitsanthony commented Jul 7, 2017

ahoma-paymentus commented Jul 9, 2017

ahoma-paymentus commented Jul 9, 2017

gyuho commented Jul 9, 2017

ahoma-paymentus commented Jul 9, 2017

hexfusion commented Jul 9, 2017 •

edited

Loading

ahoma-paymentus commented Jul 10, 2017

heyitsanthony commented Jul 12, 2017

Running the official docker image for etcd will listen only on localhost. #8230

Running the official docker image for etcd will listen only on localhost. #8230

Comments

ahoma-paymentus commented Jul 7, 2017

gyuho commented Jul 7, 2017

ahoma-paymentus commented Jul 7, 2017

gyuho commented Jul 7, 2017

ahoma-paymentus commented Jul 7, 2017

gyuho commented Jul 7, 2017

ahoma-paymentus commented Jul 7, 2017

heyitsanthony commented Jul 7, 2017

ahoma-paymentus commented Jul 9, 2017

ahoma-paymentus commented Jul 9, 2017

gyuho commented Jul 9, 2017

ahoma-paymentus commented Jul 9, 2017

hexfusion commented Jul 9, 2017 • edited Loading

ahoma-paymentus commented Jul 10, 2017

heyitsanthony commented Jul 12, 2017

hexfusion commented Jul 9, 2017 •

edited

Loading