Workspace does not start when using One namespace per user strategy strategy

[svkr2k]

Please refer to https://github.com/eclipse/che-docs/blob/master/src/main/pages/che-7/installation-guide/proc_configuring-namespace-strategies.adoc#one-namespace-per-user-strategy

I tried to setup che for One namespace per user strategy as shown below. Also, please note that I will be needing only one workspace per user.

In che.properties:

che.infra.kubernetes.namespace=
che.infra.kubernetes.namespace.default=<username>-che
che.infra.kubernetes.namespace.allow_user_defined=true
che.infra.kubernetes.pvc.strategy=common
che.infra.kubernetes.pvc.access_mode=ReadWriteOnce
che.infra.kubernetes.pvc.wait_bound=true

In che/assembly/assembly-wsmaster-war/src/main/webapp/WEB-INF/classes/che/multiuser.properties:

che.limits.user.workspaces.count=1

In che-7.5.1/deploy/kubernetes/helm/che/values.yaml

global:
  cheWorkspacesNamespace: "<username>-che"
  cheWorkspacesPVCStrategy: "common"

In che-7.5.1/deploy/kubernetes/helm/che/templates/configmap.yaml:

  CHE_INFRA_KUBERNETES_NAMESPACE: ""
  CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: {{ .Values.global.cheWorkspacesNamespace | quote}}
  CHE_INFRA_KUBERNETES_PVC_STRATEGY: "{{ .Values.global.cheWorkspacesPVCStrategy }}"

Install che:

helm upgrade --install che --force --namespace che \
  --set global.cheDomain=mydomain.com \
  --set global.cheWorkspacesNamespace='<username>-che' \
  --set global.ingressDomain=mydomain.com \
  --set cheimage=<myregistry>/testorg/che-server:latest \
  -f /home/ubuntu/src/che-7.5.1/deploy/kubernetes/helm/che/values/tls.yaml \
  -f /home/ubuntu/src/che-7.5.1/deploy/kubernetes/helm/che/values/multi-user.yaml \
  /home/ubuntu/src/che-7.5.1/deploy/kubernetes/helm/che/

$ kubectl describe cm che -n che | grep -A 2 NAMESPACE
CHE_INFRA_KUBERNETES_NAMESPACE:
----

CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT:
----
<username>-che

In browser, after I login, i try to create my first workspace. The following is the log with errors shown:

AttachVolume.Attach succeeded for volume "pvc-5bf1584a-2217-11ea-acba-029ec662a75d" 
Container image "eclipse/che-init-plugin-broker:v0.24" already present on machine
Created container
Started container
Cleaning /plugins dir
Starting Init Plugin Broker
Container image "eclipse/che-unified-plugin-broker:v0.24" already present on machine
Created container
Started container
Unified Che Plugin Broker
List of plugins and editors to install
- eclipse/my-rest-server-plugin/0.0.1 - Debug container.
- cherxp/che-ide/7.4.0 - IDE contianer
Starting Che plugins and editor processing
Starting VS Code and Theia plugins processing
All plugins have been successfully processed
Error: Failed to run the workspace: "Failure executing: POST at: https://100.64.0.1/api/v1/namespaces/r1-che/services. Message: services "my-rest-server" already exists. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=null, kind=services, name=my-rest-server, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=services "my-rest-server" already exists, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=AlreadyExists, status=Failure, additionalProperties={})."

Here is devfile:

---
apiVersion: 1.0.0
metadata:
  name: xpress
components:
  -
    type: cheEditor
    reference: https://raw.githubusercontent.com/testreg/my-plugin-registry/master/meta.yaml
  -  
    type: chePlugin
    reference: https://raw.githubusercontent.com/testreg/my-plugin-registry/master/meta-my-rest-server.yaml

Please help me setup che. Thank you in advance.

[tolusha]

@svkr2k
Is it the same #15440 ?

[svkr2k]

Thank you, @tolusha. #15440 is related to chectl.
This issue appears when we install che using helm.

[svkr2k]

any suggestions on workaround (till this issue gets resolved) will be very helpful.

[sleshchenko]

@svkr2k https://raw.githubusercontent.com/cherxp/my-plugin-registry/master/meta-my-rest-server.yaml is not available for me. Could you provide your plugin configuration?

[svkr2k]

Thank you, @sleshchenko .

Here is the yaml file https://raw.githubusercontent.com/testreg/my-plugin-registry/master/meta-my-rest-server.yaml

apiVersion: v2
publisher: eclipse
name: my-rest-server-plugin
version: 0.0.1
type: Che Plugin
displayName: My rest Service
title: Che rest Service Plugin
description: Debug container.
icon: https://www.eclipse.org/che/images/logo-eclipseche.svg
repository: https://github.com/eclipse/che-machine-exec/
firstPublicationDate: "2019-11-06"
category: Other
spec:
  endpoints:
   -  name: "my-rest-server"
      public: true
      targetPort: 8080
      attributes:
      protocol: http
  containers:
   - name: my-rest-server
     image: "2321321321321.dkr.ecr.us-east-1.amazonaws.com/myorg1/my-rest-server:latest"
     volumes:
       - mountPath: "/home/projects"
         name: myprojects
     mountSources: true
     ports:
       - exposedPort: 8080
     memoryLimit: "2Gi"

[sleshchenko]

@svkr2k If it's suitable in your case, you can try to add discoverable: false endpoint attribute.

Proposed configuration

apiVersion: v2
publisher: eclipse
name: my-rest-server-plugin
version: 0.0.1
type: Che Plugin
displayName: My rest Service
title: Che rest Service Plugin
description: Debug container.
icon: https://www.eclipse.org/che/images/logo-eclipseche.svg
repository: https://github.com/eclipse/che-machine-exec/
firstPublicationDate: "2019-11-06"
category: Other
spec:
  endpoints:
   -  name: "my-rest-server"
      public: true
      targetPort: 8080
      attributes:
        discoverable: false
        protocol: http
  containers:
   - name: my-rest-server
     image: "879521958254.dkr.ecr.us-east-1.amazonaws.com/myorg1/my-rest-server:latest"
     volumes:
       - mountPath: "/home/projects"
         name: myprojects
     mountSources: true
     ports:
       - exposedPort: 8080
     memoryLimit: "2Gi"

More info about it you can find https://github.com/eclipse/che-plugin-registry, not sure if it contains accurate description )

[svkr2k]

Thank you, @sleshchenko , i tried with discoverable: false endpoint attribute in meta-my-rest-server.yaml

When I tried to create the workspace, i got the following error after creation of workspace pods::
Error: Failed to authenticate: "Unknown error"

The text Press F5 or click here to try again. is shown instead of the IDE UI.

[sleshchenko]

@svkr2k Seems there is some issue in your editor or integration editor + jwtproxy.
I need the information of your request to be able to help you, you can share it with HAR

Before attaching it to the issue make sure that you do not expose some private info.

[svkr2k]

Hi @sleshchenko , Thank you very much for your support !
The HAR (edited) file is attached.

che-che.mydomain.com.har.txt

Here is the output from Chrome debugger console

VM264 loader.js:345 Error: Failed to authenticate: "Unknown error"
    at XMLHttpRequest.request.onreadystatechange (VM236 loader.js:295)
(anonymous) @ VM264 loader.js:345
async function (async)
(anonymous) @ VM264 loader.js:337
(anonymous) @ VM264 loader.js:349

VM264 loader.js:288 OPTIONS https://serveri8keipv9-jwtproxy-server-4400.mydomain.com/jwt/auth net::ERR_CERT_AUTHORITY_INVALID
(anonymous) @ VM264 loader.js:288
asyncAuthenticate @ VM264 loader.js:283
(anonymous) @ VM264 loader.js:341
async function (async)
(anonymous) @ VM264 loader.js:337
(anonymous) @ VM264 loader.js:349

keycloak.js:456 [KEYCLOAK] Refreshing token: token expired

loader.js:197 GET https://che-che.mydomain.com/api/workspace/workspaceif4hs458ckghkmr9 401
(anonymous) @ loader.js:197
Promise.then (async)
(anonymous) @ loader.js:196
asyncGetWorkspace @ loader.js:193
(anonymous) @ loader.js:338
async function (async)
(anonymous) @ loader.js:337
(anonymous) @ loader.js:349

loader.js:345 Error: Failed to get the workspace: "Unknown error"
    at XMLHttpRequest.xhr.onreadystatechange (VM236 loader.js:204)
(anonymous) @ loader.js:345
async function (async)
(anonymous) @ loader.js:337
(anonymous) @ loader.js:349

keycloak.js:484 [KEYCLOAK] Token refreshed
keycloak.js:831 [KEYCLOAK] Estimated time difference between browser and server is 0 seconds

loader.js:221 Uncaught DOMException: Failed to execute 'setRequestHeader' on 'XMLHttpRequest': The object's state must be OPENED.
    at Object.successCallback (https://che-che.mydomain.com/_app/loader.js:221:25)
    at Object.setSuccess (https://keycloak-che.mydomain.com/auth/js/keycloak.js:1030:27)
    at XMLHttpRequest.req.onreadystatechange (https://keycloak-che.mydomain.com/auth/js/keycloak.js:494:43)

[sleshchenko]

@svkr2k We think this commit should solve your issue 6471205
What is Che Version you use? Are you able to use nightly?

[svkr2k]

Thank you very much ! So far, i was using 7.5.1.

I tried nightly build as you suggested:
When I tried to create the workspace, i got the same error after creation of workspace pods::
Error: Failed to authenticate: "Unknown error"

[sleshchenko]

@svkr2k Hello, is your issue fixed now or you lost hope to get help?