Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement incremental daemon loading #559

Closed
wants to merge 10 commits into from

Conversation

jonjohnsonjr
Copy link
Collaborator

WIP

Fixes #205

Continuation of this PR:
google#209

This should be considered a relatively advanced option, but for folks that know
what they are doing you can reduce the amount of data that you need to encode in
the tarball for the daemon to load it.

The ultimate use case of this option will be from daemon.Write, which
currently uses the docker load interface to pull image into the daemon,
however, this currently reuploads (and redownloads) the base image on each write
in context like ko. If we can determine the set of layers that already exist
in the daemon we can elide these from the tarball to dramatically improve
performance.

Related: google#205
@codecov-io
Copy link

codecov-io commented Oct 3, 2019

Codecov Report

Merging #559 into master will decrease coverage by 0.75%.
The diff coverage is 38%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #559      +/-   ##
==========================================
- Coverage   72.57%   71.82%   -0.76%     
==========================================
  Files          94       95       +1     
  Lines        4241     4323      +82     
==========================================
+ Hits         3078     3105      +27     
- Misses        769      813      +44     
- Partials      394      405      +11
Impacted Files Coverage Δ
pkg/v1/daemon/write.go 31.11% <26.08%> (-18.89%) ⬇️
pkg/v1/tarball/write.go 58.65% <50%> (-2.18%) ⬇️
pkg/v1/tarball/options.go 84.61% <84.61%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff1ac7f...1ca5686. Read the comment docs.

@jonjohnsonjr
Copy link
Collaborator Author

Running the POC:

$ time go run main.go 
Loaded image: example.com/ubuntu:latest========================================>]     163B/163B
Loaded image: example.com/ubuntu:latest-layer_0_probe
Loaded image: example.com/ubuntu:latest-layer_1_probe
Loaded image: example.com/ubuntu:latest-layer_2_probe
Loaded image: example.com/ubuntu:latest-layer_3_probe
Loaded image: example.com/ubuntu:latest

real	0m4.672s
user	0m1.673s
sys	0m0.529s


$ time go run main.go 
Loaded image: example.com/ubuntu:latest-layer_0_probe
Loaded image: example.com/ubuntu:latest-layer_1_probe
Loaded image: example.com/ubuntu:latest-layer_2_probe
Loaded image: example.com/ubuntu:latest-layer_3_probe
Loaded image: example.com/ubuntu:latest
Loaded image: example.com/ubuntu:latest-layer_0_probe
Loaded image: example.com/ubuntu:latest-layer_1_probe
Loaded image: example.com/ubuntu:latest-layer_2_probe
Loaded image: example.com/ubuntu:latest-layer_3_probe
Loaded image: example.com/ubuntu:latest

real	0m3.265s
user	0m1.467s
sys	0m0.401s

@github-actions
Copy link

This Pull Request is stale because it has been open for 90 days with
no activity. It will automatically close after 30 more days of
inactivity. Reopen with /reopen. Mark as fresh by adding the
comment /remove-lifecycle stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

daemon.Write implementation is pretty naive
2 participants