Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New Networking wont resolve hostnames in time #2614

Closed
cescoferraro opened this issue Jan 6, 2016 · 5 comments
Closed

New Networking wont resolve hostnames in time #2614

cescoferraro opened this issue Jan 6, 2016 · 5 comments
Labels
area/networking

Comments

@cescoferraro
Copy link

Hi guys,

I have been using the -link option without any problems, until I got the time to carefully read the docs and realized that this a deprecated solution. So a decided to update one of my apps to the new networking system. My compose file contains 1 nginx and 2 webapp containers (alfa and beta that I use to deploy with minimum downtime using Ansible).

alfa:
  build: .
  net: "portifoliocesco"
  volumes:
    - ${PWD}/webapp:/srv/app
    - ${PWD}/supervisor/:/etc/supervisor/

beta:
  build: .
  net: "portifoliocesco"
  volumes:
    - ${PWD}/webapp:/srv/app
    - ${PWD}/supervisor/:/etc/supervisor/


nginx:
  image: nginx
  container_name: portifoliocesco_nginx
  net: "portifoliocesco"
  ports:
    - "9000:80"
  volumes:
    - ${PWD}/nginx/:/etc/nginx/
    - ${PWD}/webapp/app/static/:/var/www/static/
  environment:
    - VIRTUAL_HOST=www.cescoferraro.xyz

But when I run docker-compose up, NGINX thorows:

nginx_1 | 2016/01/06 22:53:48 [emerg] 1#1: host not found in upstream "portifoliocesco_alfa_1:9999" in /etc/nginx/nginx.conf:21
nginx_1 | nginx: [emerg] host not found in upstream "portifoliocesco_alfa_1:9999" in /etc/nginx/nginx.conf:21

But if I run afterwards

docker run -it --rm --net=portifoliocesco busybox wget -qO- http://portifoliocesco_alfa_1:9999  
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
....
</html>

and I can also up the NGINX container separately later too

docker-compose up nginx                          
Starting portifoliocesco_nginx
Attaching to portifoliocesco_nginx
nginx_1 | 192.168.99.1 - - [06/Jan/2016:22:57:10 +0000]  "GET / HTTP/1.1" 200 1438
@dnephin
Copy link

dnephin commented Jan 6, 2016

That's right, with the new networking there is no longer any guarantee about the order containers are started.

@cescoferraro
Copy link
Author

@dnephin any workarounds besides upping them separately?
Maybe compose could deal wth that with some sort of priority option.

@ThatRendle
Copy link

This is still being discussed extensively in #374. There are a lot of difficulties with trying to implement a general solution to the problem, but it seems to mainly come down to: what constitutes a container being up, given that when the entrypoint command exits, the container is stopped?

In the meantime, I've been working around the problem by using a Bash script with a long-enough sleep as the entrypoint for containers that depend on other containers.

@cescoferraro
Copy link
Author

For anyone that might end up here, here is a ansible way to circunvent this problem.

- name: Check if alfa and beta are up
  shell: /usr/bin/docker ps | grep "{{ project_name }}" | awk '{print $NF}' | xargs
  register: both_running
  until: '"alfa" in "{{both_running.stdout}}" and "beta" in "{{both_running.stdout}}"'
  retries: 10
  delay: 10

- fail: msg="Something bad happened!"
  when: '"alfa" not in "{{both_running.stdout}}" or "beta" not in "{{both_running.stdout}}"'

@dnephin dnephin mentioned this issue Jan 17, 2016
Closed
@dnephin
Copy link

dnephin commented Jan 17, 2016

I've created #2682 to track the issue of startup order, however I don't think that fully resolves the issue as discussed in #374.

Best practice is for the application to not assume that all the dependencies are available when it starts, and to retry any unavailable connections on every request. Unfortunately nginx doesn't really do this, so it requires some additional scripting to wait for dependencies before nginx start.

Since we already have issues tracking this, I'm going to close this one.

@dnephin dnephin closed this as completed Jan 17, 2016
@dnephin dnephin mentioned this issue Jan 17, 2016
Closed
@greylurk greylurk mentioned this issue Jan 18, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/networking
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ThatRendle @dnephin @cescoferraro