Update Kaniko to 0.24.0, use new snapshotMode

[SaschaSchwarze0]

Kaniko released v0.24.0 last week. I am updating the cluster build strategy to use the new version. I am also changing the argument set to use --snapshotMode=redo. Kaniko v0.24.0 introduces two new performance-related command line flags that are described here: Kaniko Performance Testing.

We assessed this internally, copying my assessment here:

The problem that the enhancement address go directly against the biggest issue that Kaniko has: Kaniko works by extracting the FROM image specified in the Dockerfile into its (otherwise empty) root file system. Then it does through all the commands in the Dockerfile and performs them. Some of these commands make changes to the file system. In those cases, Kaniko needs to build the appropriate layer with the delta. For ADD and COPY, it is an easy job because Kaniko knows which files it is copying around. The difficulty is RUN. This command can run arbitrary things like installations that add or modify files at any place. To determine changes, Kaniko does file system snapshotting. The comparison of two snapshots allows to see what changed.

The default snapshot mode (full) includes the hash of the file content which is expensive. There already existed the time mode that only checked mtime and is probably very fast, but has a problem in its design (see https://github.com/GoogleContainerTools/kaniko#mtime-and-snapshotting).

The new redo mode tries it more efficiently. It rents ideas from build systems that - to only build deltas - also need to know which files have changed. Specifically, the redo tool is mentioned. The file hash then does not anymore include the file content but not only the mtime.

I think it is obvious that an algorithm that had to read the file content is way more expensive than anything else that can live without it. What is good is that the proposal uses an algorithm that has been in use elsewhere already which makes me give it the trust that it works relyably.

--

The second flag, --use-new-run=true, I find critical. It tries to live without any snapshots. Instead, before a RUN command starts, it writes a file and captures its mtime. Then it runs everything from the RUN command. After that, it iterates all the files and assumes that all files with an mtime after the captured have changed or are new. Conceptually I see there three issues:

• If a file gets deleted by some RUN command, then it is not there anymore. How would that be handled? I see there some logic related to deletion in the PR, add a new run command along with a new flag GoogleContainerTools/kaniko#1300, but have not understood it so far.
• It relies on mtime and inherits the issue of the time snapshot mode. That issue is that the mtime of the inode is not immediately updated when the file is written but rather with some delay. I think that is why the PR adds a delay of one second, https://github.com/GoogleContainerTools/kaniko/pull/1300/files#diff-88867d807e8c3ae46e3a37f664fe8590R52-R53. But that feels dirty.
• There are ways to explicitly set the mtime to some value, using touch for example (unlikely to be an issue for us in our Dockerfile builds), but also tar on extraction does this (according to https://apenwarr.ca/log/20181113, search for Is mtime monotonically increasing? on that page). I brought that issue up as well.

--

My conclusion on that is that we are getting a great improvement in the run time here but have to be careful to still create correct images. The existing issues related to this enhancement, GoogleContainerTools/kaniko#1316 and GoogleContainerTools/kaniko#1317, both go in the direction of my assessment as they report issues with the --use-new-run=true flag. And specifically the npm install related issue is completely clear to me because npm hosts compressed tar files and extracts them into the correct node_modules location. And those files have their old mtime from when they were written before getting put into the tarball.

So, at the moment, there is no Kaniko release that contains these new flags. Once there is one, we should pick that up and I recommend to set --snapshotMode=redo in the Kaniko build strategy, but NOT --use-new-run=true. Based on how I understand those fixes, --snapshotMode=redo is the big performance improvement, --use-new-run=true in addition would NOT be another big step forward but only a small one if at all measurable.

In the meantime, there is the v0.24.0 release. I brought up my concerns on use-new-run here but have not received feedback yet.

[SaschaSchwarze0]

@otaviof I included you in the review as you will need to make the same update in your runtime image pr.

[SaschaSchwarze0]

/test 4.4-unit

[SaschaSchwarze0]

/test 4.4-unit

[zhangtbj]

/lgtm

[zhangtbj]

Hi @SaschaSchwarze0 ,

There is a conflict, pls help fix

[SaschaSchwarze0]

Okay, one second.

[SaschaSchwarze0]

@zhangtbj done.

[zhangtbj]

/lgtm

[zhangtbj]

/approve

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zhangtbj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [zhangtbj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment