How to restore workspaces when restarting Che?

[wn092943]

hi,
Steps to reproduce:

1. start che through executing bin/che.bat
   2.visit localhost:8080
   3.create a new workspace and create a simple java project
   4.stop che through closing the command shell where is executing bin/che.bat
   5.execute bin/che.bat again
   6.can't find any workspace and project when visiting localhost:8080

I find the size of file .docker/macine/macines/che/disk.vmdk increase when I create workspace, so I guess some information about workspace have been stored, so how to restore workspace when I restart che?

I run che on Wins 7, and the version of Che is 4.3.4(I try to use other version but get same result).

Thanks in advance.

[ddementieva]

Can you try starting and stopping workspace and then shutting down the server?

[wn092943]

@ddementieva I can successfully run and stop workspace if i don't stop or restart che server.
However, I also can't find any workspaces when I restart che server after starting and stopping workspace. any ideas I missed?

[ddementieva]

Can you check if there is anything in the /che/storage directory?

[wn092943]

@ddementieva There is a file preferences.json, and when I create a new workspace this file can record some contents about the workspace I created, like
\"workspacexhx8euul56epaze9\":{\"actions\":[{\"parameters\":{\"showHiddenFiles\":\"false\"}, \"id\":\"showHideHiddenFiles\"}]}}, \"recentWorkspaceId\":\"workspacexhx8euul56epaze9\". workspacexhx8euul56epaze9 is the id of workspace I created just now.
When I stop che and restart che, the content of preferences.json has no change, but the dashboard still can't show any workspaces or projects.

[ghost]

Can you give it a try with a nightly build https://install.codenvycorp.com/che/eclipse-che-nightly.zip?

[wn092943]

@eivantsov I restore workspaces successfully. Thanks for your advice.
By the way, after creating workspaces, only executed che stop command, these workspaces can be restored.
If I restart che server using che start or che run command, skiping the step that execute che stop command, workspaces aren't restored.

[ghost]

But how do you then terminate the process? Yes, I think that the server needs to be gracefully stopped for the workspaces to be saved into local storage.

[mhworth]

I have the same issue on Mac while using Docker for Mac. Perhaps the problem is that I'm not sure how to gracefully shutdown the server so the workspaces are restored. In general, it'd be nice if we didn't have to worry about graceful shutdown, as unexpected shutdowns are pretty likely on a workstation, and I'd hate to lose all my workspaces if that were to happen.

Note: I'm using the nightly build; I couldn't get the stable build to work with Docker for Mac.

Here's what I'm doing:

docker-compose.yml:

che:
   image: codenvy/che:nightly
   command: --remote:${DOCKER_VM_IP}
   restart: always
   ports:
   - "8080:8080"
   environment:
     CHE_LOCAL_CONF_DIR: /container
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /home/user/che/lib:/home/user/che/lib-copy
     - /home/user/che/workspaces:/home/user/che/workspaces
     - /home/user/che/storage:/home/user/che/storage
     - /home/user/che:/container
   container_name: che

Shell commands:

# Grabs the IP address of the Xhyve VM
export DOCKER_VM_IP=$(docker run --rm --net host alpine sh -c "ip a show eth0" | \
                    grep 'inet ' | cut -d/ -f1 | awk '{ print $2}')

# Sets the loopback alias for this IP
# You will be asked for your root password
sudo ifconfig lo0 alias $DOCKER_VM_IP

# Prints the IP address
echo $DOCKER_VM_IP

# Bring up Che
docker-compose up -d

# Create some workspaces, work, etc....

# Shut down Che
docker-compose stop

# Start it up again
docker-compose start

# No more workspaces

There is a preferences.json and workspaces are present in /home/user/che, but it's not picking them up on restart for whatever reason. Am I doing something wrong in the way I'm setting it up or something?

[TylerJewell]

Your compose configuration looks good so I doubt that the issue is related to the way that you are running or stopping the server. In other words I think this is a bug.

@agaragryi, @eivantsov - is this known? I don't see an issue for it in github or jira but thought it may have been discussed elsewhere.

[mhworth]

Thanks for looking into it, @TylerJewell !

[ghost]

@mhworth /home/user/che/storage:/home/user/che/storage this should do the trick for you. It's /storage dir that hosts those jsons.

[malcolmalex]

I've been running into the same issue as @mhworth with Docker for Mac, just using the instructions in the doc (without using docker-compose).

[mhworth]

@eivantsov Thanks for the direction. That mount is present in my docker-compose.yml file already (see the fourth mount), and I have confirmed that it works by overriding the command with bash and doing an ls on /home/user/che/storage. I see the json files, but Che just doesn't pick them up for some reason.

[ghost]

@mhworth @malcolmalex I have just checked it and I can see workspaces.json file created after I stop a workspace and stop che. If a container is stopped with docker stop command, the server shuts down and these files are not created. If you have /home/user/che/storage/workspaces/.json on your host mounted into /home/user/che/storage/workspaces.json in a container, Che WILL pick them up.

[jcpowermac]

I have only been using Che for a couple days and I think it is a great project. This issue so far has been my only real hangup. In a Docker I was thinking a modification to receive the signal when a docker stop to run che.sh stop. Then at least if the container is stopped the workspaces would still be available after a docker start. Examples
Or could the files not be written out some time before a stop? I could be missing something and if that is case please let me know.

[TylerJewell]

@jcpowermac I think we can close this issue as there was a bug that was resolved in 4.5.0. If you were running Che in a Docker container, such as for Docker for Mac / Windows, then there were scenarios where the workspace stuff (project, etc) was not being written into the mount and then available in between runs. See this PR: #1733.

Also, you may want to attempt the new Che launcher for docker. It dramatically simplifies many things.
https://eclipse-che.readme.io/v4.4/docs/usage-docker-launcher

[TylerJewell]

Everyone - I am going to close this issue due to #1733 - if for some reason this issue reappears, then please reopen for discussion.

[jcpowermac]

@TylerJewell this is still an issue. Also the documentation or your automated build in Docker hub is incorrect. There is no latest tag on the che-launcher.

After I created a workspace I chdir to /home/user/che/

[root@localhost workspaces]# cd ../storage/
[root@localhost storage]# ls -alh
total 0
drwxr-xr-x. 2 1000 1000  6 Jul 22 10:50 .
drwxr-xr-x. 5 root root 47 Jul 22 10:50 ..

Not until I stopped the workspace did I get a preference.json.

[root@localhost storage]# ls -alh
total 4.0K
drwxr-xr-x. 2 1000 1000  29 Jul 22 10:57 .
drwxr-xr-x. 5 root root  47 Jul 22 10:50 ..
-rw-r--r--. 1 root root 453 Jul 22 10:57 preferences.json

{
  "che": {
    "IdeAppState": "{\"workspaces\":{\"machinezpt64uhl9zwvu3ye\":{\"actions\":[{\"parameters\":{\"showHiddenFiles\":\"false\"}, \"id\":\"showHideHiddenFiles\"},{\"parameters\":{\"file\":\"/che-workflow-test/README.md\"}, \"id\":\"openFile\"},{\"parameters\":{\"revealPath\":\"\"}, \"id\":\"revealNode\"}]}}, \"recentWorkspaceId\":\"machinezpt64uhl9zwvu3ye\"}"
  },
  "codenvy": {
    "preference1": "value",
    "preference2": "value"
  }
}

Here I am stopping Che using the launcher.

[root@localhost storage]# docker run -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly stop
INFO: ECLIPSE CHE: STOPPING SERVER...
INFO: ECLIPSE CHE: REMOVING CONTAINER
INFO: ECLIPSE CHE: STOPPED

Still only preferences.json

[root@localhost storage]# ls
preferences.json
[root@localhost storage]# ls -alh
total 4.0K
drwxr-xr-x. 2 1000 1000  29 Jul 22 10:57 .
drwxr-xr-x. 5 root root  47 Jul 22 10:50 ..
-rw-r--r--. 1 1000 1000 371 Jul 22 10:58 preferences.json

Starting Che with launcher...

[root@localhost storage]# docker run -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly start
INFO: ECLIPSE CHE: ALREADY HAVE IMAGE codenvy/che-server:nightly
INFO: ECLIPSE CHE: CONTAINER STARTING
INFO: ECLIPSE CHE: SERVER LOGS AT "docker logs -f che-server"
INFO: ECLIPSE CHE: SERVER BOOTING...
INFO: ECLIPSE CHE: BOOTED AND REACHABLE
INFO: ECLIPSE CHE: http://localhost:8080

And now I have no workspaces or projects or my new stack I added. I don't understand why I have to manually stop the Che server for the workspace, stacks and / or project data to persist. Or not persist in this case. Like I said before I really like this project but the data needs to persisted when an operation occurs not when I stop the server.

[TylerJewell]

@jcpowermac ok lets separate the issues a bit. The preferences.json is only supposed to be written upon workspace stop. So that is expected behavior right now.

However the not saving your workspace + projects is definitely some bad juju. I am wondering if you have to always pass in -e CHE_DATA_FOLDER=/home/user/che now, and we forgot to default it. Would you mind doing a test on your computer for this? Also, can you please let us know which OS and docker package you are using (native, xhyve, boot2docker).

As for the che-launcher:nightly - that is expected right now. I added a comment in the docs saying that you have to use the nightly tag until 4.6.0 is released.

And - for the record - workspaces and projects are immediately persisted when they are created.

[jcpowermac]

Distro

[root@localhost storage]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

I have installed Docker from their repos.

[root@localhost storage]# docker --version
Docker version 1.11.2, build b9f10c9

docker run -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly info

DEBUG: ---------------------------------------
DEBUG: ---------  CHE DEBUG INFO   -----------
DEBUG: ---------------------------------------
DEBUG:
DEBUG: DOCKER_INSTALL_TYPE       = native
DEBUG:
DEBUG: CHE_SERVER_CONTAINER_NAME = che-server
DEBUG: CHE_SERVER_IMAGE_NAME     = codenvy/che-server
DEBUG:
DEBUG: CHE CONTAINER EXISTS?     YES
DEBUG: CHE CONTAINER IS RUNNING? YES
DEBUG: CHE CONTAINER IS STOPPED? NO
DEBUG: CHE SERVER IS BOOTED?     YES
DEBUG:
DEBUG: CHE_PORT                  = 8080
DEBUG: CHE_VERSION               = nightly
DEBUG: CHE_RESTART_POLICY        = no
DEBUG: CHE_USER                  = root
DEBUG: CHE_HOST_IP               = 10.53.252.100
DEBUG: CHE_LOG_LEVEL             = info
DEBUG: CHE_HOSTNAME              = localhost
DEBUG: CHE_DATA_FOLDER           = /home/user/che
DEBUG: CHE_CONF_FOLDER           = not set
DEBUG: CHE_LOCAL_BINARY          = not set
DEBUG:
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------

docker inspect che-server

    "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin",
                "LANG=C.UTF-8",
                "JAVA_HOME=/usr/lib/jvm/default-jvm/jre",
                "CHE_HOME=/home/user/che",
                "DOCKER_VERSION=1.6.0",
                "DOCKER_BUCKET=get.docker.com"
            ],

docker run -e CHE_DATA_FOLDER=/home/user/che -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly start

INFO: ECLIPSE CHE: ALREADY HAVE IMAGE codenvy/che-server:nightly
INFO: ECLIPSE CHE: CONTAINER STARTING
INFO: ECLIPSE CHE: SERVER LOGS AT "docker logs -f che-server"
INFO: ECLIPSE CHE: SERVER BOOTING...
INFO: ECLIPSE CHE: BOOTED AND REACHABLE
INFO: ECLIPSE CHE: http://localhost:8080

Added my stack, created a workspace and imported my project. Then I stopped the workspace

[root@localhost storage]# ls -alh
total 4.0K
drwxr-xr-x. 2 1000 1000  29 Jul 22 10:57 .
drwxr-xr-x. 5 root root  47 Jul 22 10:50 ..
-rw-r--r--. 1 1000 1000 559 Jul 22 12:49 preferences.json
[root@localhost storage]# cat preferences.json
{
  "che": {
    "IdeAppState": "{\"workspaces\":{\"machinezpt64uhl9zwvu3ye\":{\"actions\":[{\"parameters\":{\"showHiddenFiles\":\"false\"}, \"id\":\"showHideHiddenFiles\"},{\"parameters\":{\"revealPath\":\"\"}, \"id\":\"revealNode\"}]}, \"machinewat146gp5n4k9xr5\":{\"actions\":[{\"parameters\":{\"showHiddenFiles\":\"false\"}, \"id\":\"showHideHiddenFiles\"},{\"parameters\":{\"revealPath\":\"\"}, \"id\":\"revealNode\"}]}}, \"recentWorkspaceId\":\"machinewat146gp5n4k9xr5\"}"
  },
  "codenvy": {
    "preference1": "value",
    "preference2": "value"
  }
}

Stopped the container via the launcher.

[root@localhost storage]# docker run -e CHE_DATA_FOLDER=/home/user/che -v /var/run/docker.sock:/var/run/docker.sock codenvy/che-launcher:nightly stop
INFO: ECLIPSE CHE: STOPPING SERVER...
INFO: ECLIPSE CHE: REMOVING CONTAINER
INFO: ECLIPSE CHE: STOPPED
[root@localhost storage]# ls -alh
total 4.0K
drwxr-xr-x. 2 1000 1000  29 Jul 22 10:57 .
drwxr-xr-x. 5 root root  47 Jul 22 10:50 ..
-rw-r--r--. 1 1000 1000 559 Jul 22 12:49 preferences.json

docker inspect after adding -e

            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin",
                "LANG=C.UTF-8",
                "JAVA_HOME=/usr/lib/jvm/default-jvm/jre",
                "CHE_HOME=/home/user/che",
                "DOCKER_VERSION=1.6.0",
                "DOCKER_BUCKET=get.docker.com"
            ],

and launcher info

DEBUG: ---------------------------------------
DEBUG: ---------  CHE DEBUG INFO   -----------
DEBUG: ---------------------------------------
DEBUG:
DEBUG: DOCKER_INSTALL_TYPE       = native
DEBUG:
DEBUG: CHE_SERVER_CONTAINER_NAME = che-server
DEBUG: CHE_SERVER_IMAGE_NAME     = codenvy/che-server
DEBUG:
DEBUG: CHE CONTAINER EXISTS?     YES
DEBUG: CHE CONTAINER IS RUNNING? YES
DEBUG: CHE CONTAINER IS STOPPED? NO
DEBUG: CHE SERVER IS BOOTED?     YES
DEBUG:
DEBUG: CHE_PORT                  = 8080
DEBUG: CHE_VERSION               = nightly
DEBUG: CHE_RESTART_POLICY        = no
DEBUG: CHE_USER                  = root
DEBUG: CHE_HOST_IP               = 10.53.252.100
DEBUG: CHE_LOG_LEVEL             = info
DEBUG: CHE_HOSTNAME              = localhost
DEBUG: CHE_DATA_FOLDER           = /home/user/che
DEBUG: CHE_CONF_FOLDER           = not set
DEBUG: CHE_LOCAL_BINARY          = not set
DEBUG:
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------
DEBUG: ---------------------------------------

[TylerJewell]

@jcpowermac - thank you for the full analysis. Everything in your execution and run looks perfect. The workpaces and their project files should be in /home/user/che. I don't have a linux box around to test - all of my testing has been either Docker for Mac / Windows, which is usually the broken use case. So it's surprising to see a Linux configuration not saving.

This will sound pedantic for asking since you were so thorough in the reporting, but I only see /home/user/che/storage/preferences.json. You did check /home/user/che/workspaces` and it was not existent?

@l0rd, @eivantsov, @fbenoit - we may need some more eyes on this.

[TylerJewell]

@jcpowermac - I have spent the past 6 hours attempting to debug some of this. I launched the same OS version in Vagrant. I am getting a variety of different errors, for which I suspect firewalld / iptables are causing the problem.

After a 7th hour, I was able to overcome that issue :). Ok, I started with:
1: CentOS 7.2 - matches your version.
2: Installed Docker 1.11.2 - matches your version.
3: Disabled iptables and firewalld, as there are issues that firewalld creates for running Docker.
4: I then launched Che in the default configuration and ran into some problems as my Che was inside of a VM with an IP address. So I needed to launch the Che system with the following:

docker run -v /var/run/docker.sock:/var/run/docker.sock 
  -e CHE_HOST_IP=192.168.56.110 
  codenvy/che-launcher:nightly start

where CHE_HOST_IP is the external IP address that clients are accessing my VM where CentOS is installed.

5: I then created a workspace and a project.

[root@dev node-js-9pp0]# pwd
/home/user/che/workspaces/wksp-3ri5/node-js-9pp0

[root@dev node-js-9pp0]# ls -al
total 16
drwxr-xr-x. 5 vagrant vagrant  101 Jul 24 22:32 .
drwxr-xr-x. 3 vagrant root      25 Jul 24 22:32 ..
drwxr-xr-x. 2 vagrant vagrant   19 Jul 24 22:32 app
-rw-r--r--. 1 vagrant vagrant 2027 Jul 24 22:32 .codenvy.json
drwxr-xr-x. 7 vagrant vagrant 4096 Jul 24 22:32 .git
drwxr-xr-x. 3 vagrant vagrant   20 Jul 24 22:32 node_modules
-rw-r--r--. 1 vagrant vagrant  259 Jul 24 22:32 package.json
-rw-r--r--. 1 vagrant vagrant  449 Jul 24 22:32 README.md

All of my files from the project are located in the directory and their are editable by the vagrant user.

[jcpowermac]

@TylerJewell I will test again today. Did you disable SELinux? Just want to confirm so that we are more or less configured in the same way.

[TylerJewell]

@jcpowermac - I didn't check. I just took the default centos vagrant image. I presume selinux is not disabled in the default configuration?

[jcpowermac]

By default its enabled

[jcpowermac]

@TylerJewell same result. The one constant though is my stack, could that be causing this problem? I will retest with a built in stack to confirm.

Step 1 - Using che-launcher

Step 2 - Add Stack

> $ curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d @che-ansible.json http://10.53.252.100:8080/api/stack

{"name":"Ansible 2.1","source":{"origin":"jcpowermac/eclipse-che-ansible","type":"image"},"components":[{"version":"2.1","name":"Ansible"},{"version":"2.7.11","name":"Python"},{"version":"8.1.1","name":"pip"}],"tags":["Python 2.7","pip","Ansible","Git"],"id":"stackqwtmzal3q7izbqqz","workspaceConfig":{"environments":[{"name":"default","machineConfigs":[{"limits":{"ram":0},"name":"default","type":"docker","source":{"location":"stub","type":"dockerfile"},"dev":true,"servers":[],"envVariables":{},"links":[]}]}],"defaultEnv":"default","projects":[],"name":"default","commands":[{"commandLine":"cd ${current.project.path} && ansible-playbook -vvv site.yml","name":"run","attributes":{"previewUrl":"http://${server.port.8080}"},"type":"custom"}],"links":[]},"creator":"che","description":"Default Ansible Stack with Ansible 2.1","scope":"advanced","links":[{"href":"http://10.53.252.100:8080/wsmaster/api/stack/stackqwtmzal3q7izbqqz","parameters":[],"rel":"remove stack","method":"DELETE"},{"href":"http://10.53.252.100:8080/wsmaster/api/stack/stackqwtmzal3q7izbqqz","parameters":[],"rel":"get stack by id","method":"GET","produces":"application/json"}]}%

Step 3 - Create workspace and project

Step 4 - Screenshot of workspace and project

Step 5 - Stop the workspace

Step 6 - Tree, Docker status

[TylerJewell]

@jcpowermac - first, I vote that all support issues should come with ascii animations. That is crazy cool.

But in watching your second video, what I saw was that the workspace storage is happening properly. Your project files were saved to disk. You can see them at the beginning where you did the tree and at the end.

So, if you restarted the Che server, you should see the workspace that you had already created and be able to start it again. And when it is restarted, the project files will be loaded from disk. FYI, I believe that nightly, for which you are running 4.6.0-RC1-SNAPSHOT, has a known issue around restarting workspaces from snapshotted images. But the key point was that earlier you were concerned that your project files are not being saved in between runs, and it looks like they are.

Now, workspace image snapshots - those are saved in a different location - were you wanting to track that down?

[jcpowermac]

@TylerJewell sorry maybe I didn't make the issue clear. I have tried multiple versions of che:

• 4.4.1
• 4.5
• 4.6

The issue that I am having is if I ever stop the che server and restart it the workspaces or projects are no longer available via the UI. The workspace directory structure and the projects beneath are always available.

My use case for Che was to provide an environment where a typical Windows administrator can write and run Ansible playbooks/roles without messing with installing/configuring Linux. I would provide a template for VMware vSphere that can be imported.

[TylerJewell]

Ahh, @jcpowermac - my apologies, I did misunderstand the issue. Is it possible that this issue is related to #1902 and #1919? These issues have been discovered on the 4.6 nightlies. I am not sure if anyone has confirmed them on 4.5 / 4.4.1 - so you may be experiencing something else. If those other issues look similar, then this may resolve itself really quickly. If not, then I may want to ask for a diagnostic session on your system.

Are there any exceptions or error messages in your browser console (CTRL-SHIFT-I) when starting the workspace?

Clearly your issue is a real issue and fundamental to the product, so if this is a bug, it will get a blocker designation.

[jcpowermac]

@TylerJewell I tested #1902 or #1919 and did not experience that issue. I will re-test today and provide feedback.

[TylerJewell]

@jcpowermac - we have found a blocker issue with how workspaces are stopped from the launcher. #1949.

[TylerJewell]

@jcpowermac and others - this has been merged into master. The pull request is here.
#1956