Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Workspace agent not starting when CHE_SINGLE_PORT=true #5240

Closed
levjason opened this issue May 30, 2017 · 17 comments
Closed

Workspace agent not starting when CHE_SINGLE_PORT=true #5240

levjason opened this issue May 30, 2017 · 17 comments
Assignees
@benoitf
Labels
kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Milestone
5.12.0

Comments

@levjason
Copy link

levjason commented May 30, 2017 •
edited
Loading

I enabled CHE_SINGLE_PORT in che.env. The traefik proxy starts. But none of the workspaces will run.

When a workspace is started I notice in the browser debugger:
_app-0.js:851 GET http://wsagent.dev-machine.workspacei121v3s3086c91js.172.17.0.1.nip.io:8080/api/ net::ERR_CONNECTION_TIMED_OUT

Reproduction Steps:

  1. Add CHE_SINGLE_PORT=true to che.env.
  2. Start che
  3. Try to run a workspace

OS and version:
Ubuntu Linux, 16.04.2

Diagnostics:

INFO: (che cli): 5.11.2 - using docker 1.13.1 / native

CLI:
TTY: true
Daemon: /var/run/docker.sock
Image: eclipse/che:5.11.2
Version: 5.11.2
Command: info
Parameters: info
Mounts:
/data: /home/usajlev/che_data
/data/instance: not set
/data/backup: not set
/repo: not set
/assembly: not set
/sync: not set
/unison: not set
/chedir: not set
System:
Docker: native
Proxy: not set
Internal:
CHE_VERSION: 5.11.2
CHE_HOST: 10.130.54.161
CHE_INSTANCE: /home/usajlev/che_data/instance
CHE_CONFIG: /home/usajlev/che_data
CHE_BACKUP: /home/usajlev/che_data/backup
CHE_REGISTRY: /version
CHE_DEBUG: false
IP Detection: 10.130.54.161
Initialized: true
Image Registry:
IMAGE_INIT=eclipse/che-init:5.11.2
IMAGE_CHE=eclipse/che-server:5.11.2
IMAGE_COMPOSE=docker/compose:1.8.1
IMAGE_TRAEFIK=traefik:v1.3.0-rc1
BOOTSTRAP_IMAGE_ALPINE=alpine:3.4
BOOTSTRAP_IMAGE_CHEIP=eclipse/che-ip:5.11.2
UTILITY_IMAGE_CHEACTION=eclipse/che-action:5.11.2
UTILITY_IMAGE_CHEDIR=eclipse/che-dir:5.11.2
UTILITY_IMAGE_CHETEST=eclipse/che-test:5.11.2
UTILITY_IMAGE_CHEMOUNT=eclipse/che-mount:5.11.2
che.env:
CHE_HOST=10.130.54.161
CHE_SINGLE_PORT=true
CHE_DOCKER_SERVER__EVALUATION__STRATEGY_CUSTOM_TEMPLATE=...:
@ghost ghost added kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system. labels May 31, 2017
@ghost
Copy link

ghost commented May 31, 2017

@levjason @benoitf I have got slightly different results. In my case, the IDE can kind of connect to ws agent, but not entirely:

image

My traefik container logs - https://gist.githubusercontent.com/eivantsov/5eeac19f7aa94de466ad31714cf39f07/raw/0f7ee53f8e3fc229c7c87f134a25d0c44debb5b2/traefik

I pulled the latest nightly image.

@benoitf benoitf self-assigned this May 31, 2017
@ghost
Copy link

ghost commented May 31, 2017

The problem is that when IDE tries to reach ws agent, ws-master APIs are returned.

@benoitf
Copy link
Contributor

benoitf commented May 31, 2017 •
edited
Loading

Hello, I have done some investigation.

The issue is that routes created for traefik are on the format
terminal.dev-machine.workspacec9foemo8xbg41034.172.16.148.171.nip.io:8080 (public ip address) while links provided to the client are on the format terminal.dev-machine.workspacec9foemo8xbg41034.172.17.0.1.nip.io:8080

(the ip before nip.io differs)

as workaround,

when you start che, it displays at the end

INFO: (che start): Booted and reachable
INFO: (che start): Ver: nightly
INFO: (che start): Use: http://172.16.148.171:8080
INFO: (che start): API: http://172.16.148.171:8080/swagger
INFO: (che start): Debug: http://172.16.148.171:8000

so you can run che by providing this IP to CHE_DOCKER_IP_EXTERNAL and CHE_IP:

$ docker run ... -e CHE_SINGLE_PORT=true -e CHE_DOCKER_IP_EXTERNAL=172.16.148.171 -e CHE_IP=172.16.148.171 .... eclipse/che:nightly start

or you run che by customizing the template

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock -e CHE_SINGLE_PORT=true -e CHE_DOCKER_SERVER__EVALUATION__STRATEGY_CUSTOM_TEMPLATE="<serverName>.<machineName>.<workspaceId>.172.16.148.171.nip.io:8080" .... eclipse/che:nightly start

@levjason
Copy link
Author

Hello,

The workspaces start with the workaround. But I noticed these messages in the browser console:

_app-0.js:15281 WARNING (org.eclipse.che.ide.websocket.impl.BasicWebSocketEndpoint): Error occurred for endpoint ws://exec-agent.dev-machine.workspaceu777zm5jkrsw0wpj.11.111.11.111.nip.io:8080/connect
Uqj @ _app-0.js:15281
Wqj @ _app-0.js:15283
Vqj @ _app-0.js:15282
_qj @ _app-0.js:15288
Hsj @ _app-0.js:15351
onerror @ _app-0.js:15382
_app-0.js:15382 WebSocket connection to 'ws://exec-agent.dev-machine.workspaceu777zm5jkrsw0wpj.11.111.11.111.nip.io:8080/connect' failed: Error during WebSocket handshake: Unexpected response code: 504
ptj @ _app-0.js:15382
Osj @ _app-0.js:15355
Yk @ _app-0.js:280
zfl @ _app-0.js:277
hq @ _app-0.js:442
kq @ _app-0.js:445
(anonymous) @ _app-0.js:444
_app-0.js:15281 WARNING (org.eclipse.che.ide.websocket.impl.BasicWebSocketEndpoint): Error occurred for endpoint ws://exec-agent.dev-machine.workspaceu777zm5jkrsw0wpj.11.111.11.111.nip.io:8080/connect
Uqj @ _app-0.js:15281
Wqj @ _app-0.js:15283
Vqj @ _app-0.js:15282
_qj @ _app-0.js:15288
Hsj @ _app-0.js:15351
onerror @ _app-0.js:15382

@ghost
Copy link

ghost commented Jun 1, 2017

@levjason have you started an existing or a new workspace? I had issues with an existing ws. Starting a new fixed it.

@levjason
Copy link
Author

levjason commented Jun 1, 2017 •
edited by ghost
Loading

@eivantsov
I have the workaround applied. I created a new workspace. But today, I can't get the workspace to start. These are messages from the browser console:

_app-0.js:851 GET http://wsagent.dev-machine.workspacez90fl52koe71awwl.11.111.11.111.nip.io:8080/api/ 504 (Gateway Timeout)
RB @ _app-0.js:851
SB @ _app-0.js:852
vIi @ _app-0.js:13891
uIi @ _app-0.js:13890
yIi @ _app-0.js:13894
Poe @ _app-0.js:4447
(anonymous) @ _app-0.js:4450
Noe @ _app-0.js:4445
Dwf @ _app-0.js:6557
pJf @ _app-0.js:6995
(anonymous) @ _app-0.js:4456
_app-0.js:851 GET http://wsagent.dev-machine.workspacez90fl52koe71awwl.11.111.11.111.nip.io:8080/api/ 504 (Gateway Timeout)
RB @ _app-0.js:851
SB @ _app-0.js:852
vIi @ _app-0.js:13891
uIi @ _app-0.js:13890
yIi @ _app-0.js:13894
Poe @ _app-0.js:4447
(anonymous) @ _app-0.js:4450
Noe @ _app-0.js:4445
Dwf @ _app-0.js:6557
pJf @ _app-0.js:6995
(anonymous) @ _app-0.js:4456
:8080/che/wksp-aNewOne:1 XMLHttpRequest cannot load http://wsagent.dev-machine.workspacez90fl52koe71awwl.11.111.11.111.nip.io:8080/api/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://myserver29:8080' is therefore not allowed access. The response had HTTP status code 504.
_app-0.js:851 GET http://wsagent.dev-machine.workspacez90fl52koe71awwl.11.111.11.111.nip.io:8080/api/languageserver/supported 504 (Gateway Timeout)
RB @ _app-0.js:851
SB @ _app-0.js:852
vIi @ _app-0.js:13891
uIi @ _app-0.js:13890
DIi @ _app-0.js:13896
Poe @ _app-0.js:4447
(anonymous) @ _app-0.js:4450
Noe @ _app-0.js:4445
g7j @ _app-0.js:16818
o$j @ _app-0.js:16531
hJf @ _app-0.js:6991
pJf @ _app-0.js:6995
(anonymous) @ _app-0.js:4456
_app-0.js:851 GET http://wsagent.dev-machine.workspacez90fl52koe71awwl.11.111.11.111.nip.io:8080/api/languageserver/supported 504 (Gateway Timeout)
RB @ _app-0.js:851
SB @ _app-0.js:852
vIi @ _app-0.js:13891
uIi @ _app-0.js:13890
DIi @ _app-0.js:13896
Poe @ _app-0.js:4447
(anonymous) @ _app-0.js:4450
Noe @ _app-0.js:4445
g7j @ _app-0.js:16818
o$j @ _app-0.js:16531
hJf @ _app-0.js:6991
pJf @ _app-0.js:6995
(anonymous) @ _app-0.js:4456
:8080/che/wksp-aNewOne:1 XMLHttpRequest cannot load http://wsagent.dev-machine.workspacez90fl52koe71awwl.11.111.11.111.nip.io:8080/api/languageserver/supported. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://myserver29:8080' is therefore not allowed access. The response had HTTP status code 504.
_app-0.js:15281 ERROR (org.eclipse.che.plugin.languageserver.ide.LanguageServerFileTypeRegister): 
java.lang.Exception: 
	at Unknown.hp(Throwable.java:74)
	at Unknown.pp(Exception.java:33)
	at Unknown.s$j(LanguageServerFileTypeRegister.java:133)
	at Unknown.u$j(LanguageServerFileTypeRegister.java:132)
	at Unknown.epe(JsPromise.java:43)
	at Unknown.anonymous(JsPromise.java:166)
	at Unknown.lB(SimpleEventBus.java:208)
	at Unknown.vvj(SimpleEventBus.java:88)
	at Unknown.xvj(WorkspaceEventsHandler.java:143)
	at Unknown.anonymous(JsPromise.java:116)
Caused by: org.eclipse.che.ide.commons.exception.ServerDisconnectedException: undefined
	at Unknown.np(Throwable.java:61)
	at Unknown.dag(ServerDisconnectedException.java:22)
	at Unknown.pjh(AsyncRequestCallback.java:203)
	at Unknown.wjh(AsyncRequestCallback.java:93)
	at Unknown.HB(Request.java:250)
	at Unknown.$B(RequestBuilder.java:412)
	at Unknown.anonymous(XMLHttpRequest.java:329)
	at Unknown.hq(Impl.java:239)
	at Unknown.kq(Impl.java:291)
	at Unknown.anonymous(Impl.java:77)
	at Unknown.lB(SimpleEventBus.java:208)
	at Unknown.vvj(SimpleEventBus.java:88)
	at Unknown.xvj(WorkspaceEventsHandler.java:143)
	at Unknown.anonymous(JsPromise.java:116)
Uqj @ _app-0.js:15281
Wqj @ _app-0.js:15283
Vqj @ _app-0.js:15282
Zqj @ _app-0.js:15286
zJf @ _app-0.js:7001
s$j @ _app-0.js:16533
u$j @ _app-0.js:16535
epe @ _app-0.js:4461
(anonymous) @ _app-0.js:4452
_app-0.js:15281 ERROR (org.eclipse.che.ide.websocket.impl.WebSocketPropertyManager): Connection is not properly initialized, no properties set. Call 'initializedConnection' first
Uqj @ _app-0.js:15281
Wqj @ _app-0.js:15283
Vqj @ _app-0.js:15282
Zqj @ _app-0.js:15286
qtj @ _app-0.js:15383
stj @ _app-0.js:15385
jtj @ _app-0.js:15376
TXh @ _app-0.js:12182
dtf @ _app-0.js:6423
yxf @ _app-0.js:6582
Axf @ _app-0.js:6584
lB @ _app-0.js:830
kvj @ _app-0.js:15457
UIf @ _app-0.js:6981
WIf @ _app-0.js:6983
lB @ _app-0.js:830
Zxj @ _app-0.js:15565
ayj @ _app-0.js:15568
r3d @ _app-0.js:3878
W2d @ _app-0.js:3864
R2d @ _app-0.js:3860
r2d @ _app-0.js:3838
tXh @ _app-0.js:12156
s2d @ _app-0.js:3839
gtj @ _app-0.js:15373
Isj @ _app-0.js:15351
onmessage @ _app-0.js:15382
_app-0.js:402 Uncaught Error: com.google.web.bindery.event.shared.UmbrellaException: Exception caught: Exception caught: Connection is not properly initialized, no properties set. Call 'initializedConnection' first
    at DB.ip [as zn] (_app-0.js:402)
    at DB.lp [as Bn] (_app-0.js:402)
    at DB.hp (_app-0.js:400)
    at DB.pp (_app-0.js:404)
    at DB.tp (_app-0.js:408)
    at DB (_app-0.js:842)
    at lB (_app-0.js:830)
    at Zxj (_app-0.js:15565)
    at _xj.ayj [as Cr] (_app-0.js:15568)
    at r3d (_app-0.js:3878)
    at W2d (_app-0.js:3864)
    at R2d (_app-0.js:3860)
    at r2d (_app-0.js:3838)
    at tXh (_app-0.js:12156)
    at s2d (_app-0.js:3839)
    at gtj (_app-0.js:15373)
    at Fsj.Isj [as RF] (_app-0.js:15351)
    at WebSocket.onmessage (_app-0.js:15382)
ip @ _app-0.js:402
lp @ _app-0.js:402
hp @ _app-0.js:400
pp @ _app-0.js:404
tp @ _app-0.js:408
DB @ _app-0.js:842
lB @ _app-0.js:830
Zxj @ _app-0.js:15565
ayj @ _app-0.js:15568
r3d @ _app-0.js:3878
W2d @ _app-0.js:3864
R2d @ _app-0.js:3860
r2d @ _app-0.js:3838
tXh @ _app-0.js:12156
s2d @ _app-0.js:3839
gtj @ _app-0.js:15373
Isj @ _app-0.js:15351
onmessage @ _app-0.js:15382

@levjason
Copy link
Author

levjason commented Jun 1, 2017

@eivantsov Actually, I tried again and this time the workspace (the one I created today) did start.

@ghost
Copy link

ghost commented Jun 1, 2017

@levjason Can you try it a few times in an incognito tab?

@levjason
Copy link
Author

levjason commented Jun 1, 2017 •
edited by ghost
Loading

@eivantsov I restarted the server. Started an incognito window.

  1. Started workspace created today - no problem
  2. Started workspace created earlier - no problem
  3. Stopped both workspaces.
  4. Started the workspace created today - no problem
  5. Started an older workspace - failed:
:8080/che/wksp-zx72:1 XMLHttpRequest cannot load http://wsagent.dev-machine.workspacei121v3s3086c91js.11.111.11.111.nip.io:8080/api/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://myserver29:8080' is therefore not allowed access. The response had HTTP status code 504.
2VM154 _app-0.js:851 GET http://wsagent.dev-machine.workspacei121v3s3086c91js.11.111.11.111.nip.io:8080/api/languageserver/supported 504 (Gateway Timeout)
RB @ VM154 _app-0.js:851
SB @ VM154 _app-0.js:852
vIi @ VM154 _app-0.js:13891
uIi @ VM154 _app-0.js:13890
DIi @ VM154 _app-0.js:13896
Poe @ VM154 _app-0.js:4447
(anonymous) @ VM154 _app-0.js:4450
Noe @ VM154 _app-0.js:4445
g7j @ VM154 _app-0.js:16818
o$j @ VM154 _app-0.js:16531
hJf @ VM154 _app-0.js:6991
pJf @ VM154 _app-0.js:6995
(anonymous) @ VM154 _app-0.js:4456
:8080/che/wksp-zx72:1 XMLHttpRequest cannot load http://wsagent.dev-machine.workspacei121v3s3086c91js.11.111.11.111.nip.io:8080/api/languageserver/supported. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://myserver29:8080' is therefore not allowed access. The response had HTTP status code 504.
VM154 _app-0.js:15281 ERROR (org.eclipse.che.plugin.languageserver.ide.LanguageServerFileTypeRegister): 
java.lang.Exception: 
	at Unknown.hp(Throwable.java:74)
	at Unknown.pp(Exception.java:33)
	at Unknown.s$j(LanguageServerFileTypeRegister.java:133)
	at Unknown.u$j(LanguageServerFileTypeRegister.java:132)
	at Unknown.epe(JsPromise.java:43)
	at Unknown.anonymous(JsPromise.java:166)
	at Unknown.lB(SimpleEventBus.java:208)
	at Unknown.vvj(SimpleEventBus.java:88)
	at Unknown.xvj(WorkspaceEventsHandler.java:143)
	at Unknown.anonymous(JsPromise.java:116)
Caused by: org.eclipse.che.ide.commons.exception.ServerDisconnectedException: undefined
	at Unknown.np(Throwable.java:61)
	at Unknown.dag(ServerDisconnectedException.java:22)
	at Unknown.pjh(AsyncRequestCallback.java:203)
	at Unknown.wjh(AsyncRequestCallback.java:93)
	at Unknown.HB(Request.java:250)
	at Unknown.$B(RequestBuilder.java:412)
	at Unknown.anonymous(XMLHttpRequest.java:329)
	at Unknown.hq(Impl.java:239)
	at Unknown.kq(Impl.java:291)
	at Unknown.anonymous(Impl.java:77)
	at Unknown.lB(SimpleEventBus.java:208)
	at Unknown.vvj(SimpleEventBus.java:88)
	at Unknown.xvj(WorkspaceEventsHandler.java:143)
	at Unknown.anonymous(JsPromise.java:116)
Uqj @ VM154 _app-0.js:15281
Wqj @ VM154 _app-0.js:15283
Vqj @ VM154 _app-0.js:15282
Zqj @ VM154 _app-0.js:15286
zJf @ VM154 _app-0.js:7001
s$j @ VM154 _app-0.js:16533
u$j @ VM154 _app-0.js:16535
epe @ VM154 _app-0.js:4461
(anonymous) @ VM154 _app-0.js:4452

@benoitf
Copy link
Contributor

benoitf commented Jun 1, 2017

hello, what is
'http://myserver29:8080' ?
Are you proxyfing che with another host ?

@levjason
Copy link
Author

levjason commented Jun 1, 2017

@benoitf Hello, the che server is running on this server. I use this URL to access the dashboard.

@benoitf
Copy link
Contributor

benoitf commented Jun 1, 2017

@levjason if you use URL : http://11.111.11.111.nip.io:8080 ?

@levjason
Copy link
Author

levjason commented Jun 1, 2017

@benoitf If I use that URL I get similar results. That is, the new workspace can be stopped and started OK. The older workspace failed to start on the first attempt. But it succeeded on the second try.

@ghost
Copy link

ghost commented Jun 1, 2017 •
edited by ghost
Loading

@benoitf by the way I had the same behavior on my Ubuntu laptop. Not sure why it happened though.

@benoitf benoitf mentioned this issue Jun 2, 2017
Merged
@benoitf benoitf added status/in-progress This issue has been taken by an engineer and is under active development. and removed status/in-progress This issue has been taken by an engineer and is under active development. labels Jun 2, 2017
@benoitf benoitf added this to the 5.12.0 milestone Jun 4, 2017
@benoitf benoitf closed this as completed Jun 4, 2017
@benoitf
Copy link
Contributor

benoitf commented Jun 4, 2017

fixed in nightly / 5.12.0

@perspectivus1
Copy link
Contributor

Installed the nightly build. The fix is there, since now the workspace agent's address that's provided to the browser contains the external Docker's IP address (just before nip.io).

I can now create a workspace and it seems to be started, but I never get to the project creation stage, and the workspace's IDE does not load. Here is the last part of the diagnostics logs:

output channel message :{"machineName":"dev-machine","content":"[DOCKER] ---> Using cache"}
[DOCKER] ---> 3d1fb16afb4c
output channel message :{"machineName":"dev-machine","content":"[DOCKER] ---> 3d1fb16afb4c"}
[ERROR] null
output channel message :{"machineName":"dev-machine","content":"[ERROR] null"}
StatusChannel : {"machineId":"machines8im54m7cticxy5k","eventType":"ERROR","dev":true,"machineName":"dev-machine","workspaceId":"workspacepb02zwb12bd6pq5d"}
Error while starting the workspace : {"machineId":"machines8im54m7cticxy5k","eventType":"ERROR","dev":true,"machineName":"dev-machine","workspaceId":"workspacepb02zwb12bd6pq5d"}

Any ideas? Do I need to change other environment variables (other than setting CHE_SINGLE_PORT to true)?

@ghost
Copy link

ghost commented Jun 8, 2017

@perspectivus1 can you post a screenshot with browser dev console on when you start a workspace?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benoitf benoitf

Labels
kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Projects
None yet
Milestone
5.12.0
Development

No branches or pull requests
3 participants
@benoitf @perspectivus1 @levjason