Prevent Che start if 'nightly' or 'latest' images are not current.

[TylerJewell]

The nightly and latest images are convenience images desigend to make lives of Che developers easier. However, there are circumstances where changes to the core nightly image for the CLI requires that all of the underlying dependency images are updated with it. Getting the most recent CLI nightly, but leaving the eclipse/che-init:nightly or eclipse/che-server:nightly as older versions can cause problems.

A savvy developer can use --pull or --force to forcibly clean this up, but when the errors occur it is not obvious that this is the approach to clean things up.

We have tried looking at the created-date within the local image cached on a computer with the matching parameter on DockerHub to see if there is a difference. However, the property from dockerhub only reflects when the image was uploaded and does not match the property on the local image, so these values cannot be used for a meaningful comparison.

In doing some tests, it is possible from within the CLI of a Docker container, to check the current date and compare it to the date that certain images were written to disk. We could therefore implement some logic such as:

1. Get the current date of the Docker engine host (Docker VMs can have dates not matching host). On Windows, while today is January 6th, my HyperV VM has the date as December 26th, so we will need to baseline dates.

2. Get the date that the eclipse/che-cli:nightly image was written to disk.

3. If the date the image is written to disk is >24 hours, then we should display the warning message.

4. We could then get the list of dependency images that Che requires (after its been loaded) and then perform a similar check for each one of those. If the dependency images were written to disk >24 hours, we could then detect that other images are older, would they like us to assert a --force where we rmi and pull the image to guarantee it is current.

Another variation of this is that any time you run a nightly or latest image we could display a table to stdout that shows all of the images loaded by Che, their name, and how many days it has been since they were written to disk with a hint that --force will repull all images. So this way any time a developer sees that some of their images are >1 day old, they know that any issues could just require a repull.

# Get the docker image id
docker image --no-trunc eclipse/che-cli:nightly

# List the particular file and gets it date written to disk
docker run -it -v /var/lib/docker:/var/lib/docker alpine ls -al /var/lib/docker/image/aufs/imagedb/content/sha256 | grep <image-id> | <cut-to-get-date>

[TylerJewell]

@benoitf @skabashnyuk @garagatyi @riuvshin - I think I have found a way that we can solve the nightly image problem. While we would need to do something like this in the CLI - a similar sort of background thread inside of Che may be needed for things like stack images and workspace images so that we build a continuous database of information about image tags, dates images were written to disk, and then using that information to make smart notifications for end users about whether they need to create a new workspace or not.

[garagatyi]

We have tried looking at the created-date within the local image cached on a computer with the matching parameter on DockerHub to see if there is a difference. However, the property from dockerhub only reflects when the image was uploaded and does not match the property on the local image, so these values cannot be used for a meaningful comparison.

Can you elaborate, I didn't take that.

Here are couple of notes to proposed solution:

• I don't see value and good UX in re-creation of workspace created from, for example, ubuntu base image. Can you elaborate on why it is useful?
• Do you think that usage of docker internals structure (like paths on local system and usage of aufs driver only) is reliable enough?

Here an alternative idea. What do you think if we use CLI digest as init and server tags? In that case CLI will be able to use only matching images of init , server, etc.
Then updating of nightly/latest CLI will cause pulling of matching images.
But what if user calls CLI --pull. Looks like this is not needed with this approach.

[TylerJewell]

Ok - I was not aware of the concept of digests. Upon reading about them this is absolutely the best solution for us. This requires us to redo how our ci systems work as we are generating the image registry only with image name and tag. We would need that registry to include the digests too. And we should do this for all images not just nightly. You are correct that the --pull would never be needed again.

[TylerJewell]

@riuvshin - in reading online about the issues with pulling in various tags, I am now concerned that this sort of tag issue could be a hidden disaster, and so treating this as an enhancement, a bug, and a blocker. We will probably need to do a 5.0.1 release that updates a few things.

Documentation on how to use digests:
https://docs.docker.com/engine/reference/commandline/pull/#/pull-an-image-by-digest-immutable-identifier

So instead of IMAGE_INIT=eclipse/che-init:latest in the /version file, it would be IMAGE_INIT=eclipse/che-init@sha256:<digest>

• We need to treat the build of eclipse/che-base as a special CI activity. Since eclipse/che-cli inherits from this base image, we need the CI system to build eclipse/che-base, retrieve the digest, and then update the /dockerfiles/cli/Dockerfile` to alter the FROM line to use a digest entry instead of a tagged entry. We would also need the CI system to use this base image digest to update other CLIs like Codenvy. We would also still have tagged versions of the images as well.
• Modify our CI systems when it generates eclipse/che-server or eclipse/che-init to retrieve the digest of a Docker image at the end of a build.
• Have the digest values for che-server and che-init added into the /version files by CI after the previous two steps.
• We can then have the CI system generate eclipse/che-cli:<tag> and this image will be built from a digest version of eclipse/che-base@<digest> and it will define the list of dependency images with the proper digest that are needed.
• We should separately implement in the CLI a way to look at hte date at which the digest of the current eclipse/che:nightly image was written to disk. If the digest was written to disk >24 hours, then we should print out the warning message that the nightly image on disk is quite old and should be repulled. Tyler can handle this one.

[riuvshin]

@TylerJewell CI can't get digest and push changes to git repo, CI can only pull & build.
even if we do hacks to make it possible in case of CHE it will be rules violation if CI system will push changes to eclipse.
As you've probably seen to make release 5.0.0 I manually set 5.0.0 tags in dockerfiles (ex: cli use 5.0.0 tag of base image) before release and after release I manually set tags to nightly (which is snapshot alternative)

[riuvshin]

Also that means that CI system will need do regular commits to repositories for nightly, that is not acceptable.

[riuvshin]

My proposal is to add cli image to images file and when we will perform --pull CLI will be pulled, then we will need think how re start CLI after pull.

[garagatyi]

Yes, it is not good way to commit by CI on each build.

[TylerJewell]

Ok, I agree that the CI approach seems like it introduces more cost than profit.

I don't think the --pull option works because it requires the user to know that they may be out of sync and to initiate such a pull.

What we could do:

1. Add eclipse/che@sha256 to the images file that is within eclipse/che:nightly image.
2. Have eclipse/che:nightly image check the currently running container to see if the current container comes from an image that matches that digest.
3. Then stop the current container with message that nightly has been updated, please restart.

[TylerJewell]

Merged to master.