Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow exporting cache metadata in the image config #777

Merged
merged 6 commits into from
Jan 27, 2019

Conversation

tonistiigi
Copy link
Member

fixes #752

based on #615

Exporting with --export-cache type=inline importing is connected with the regular registry importer that falls back to inline if the registry ref is not a special cache manifest.

TODO:

  • tests
  • currently not deterministic on local sources
  • @AkihiroSuda I'm thinking of changing the value to base64 of the protobuf and turn the existing cacheconfig types to proto (manifest list variant will remain json). maybe even store digests as bytes with a custom marshaler. thoughts?

@tonistiigi tonistiigi force-pushed the export-cache-inline branch from b2e1583 to 3dd50bb Compare January 9, 2019 02:07
@tonistiigi tonistiigi added this to the v0.4.0 milestone Jan 9, 2019
@AkihiroSuda
Copy link
Member

thoughts

SGTM

@tonistiigi
Copy link
Member Author

tonistiigi commented Jan 18, 2019

FIxed some issues, changed to base64 of json and added tests. Also added more logic for cache stability: walking back unused remote keys for exporting, keeping description on non-dockerfile, and copying matched remote record with the cache of the parent layers.

There is still a case where unused keys are not copied to the local store on match, but it is tricky to add that because of the incompatibility between remote format and internal cache keys. The remote format includes output index in digest while cache keys keep it separate meaning we can only convert one way. It should be possible to match them but keeping existing keys might be complicated. This seems low priority atm.

still based on #615 . Otherwise ready for review.

@AkihiroSuda @tiborvass

@AkihiroSuda
Copy link
Member

LGTM after rebase

Can we release v0.4.0-beta.0 after this PR gets merged?

@tonistiigi
Copy link
Member Author

@AkihiroSuda There are couple of issues still in the milestone (can you take #774?). Don't think we need betas atm unless you have some specific idea where to test it. In case of a big PR like this(or local type) lets just wait for couple of days after a merge.

digests = append(digests, desc.Digest)
}

if _, err := res.CacheKeys()[0].Exporter.ExportTo(ctx, e, solver.CacheExportOpt{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is CacheKeys() guaranteed to return more than one entries?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if multiple keys are returned they all share the exporter https://github.com/moby/buildkit/blob/master/solver/edge.go#L871

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Proposal: optionally export cache metadata in the image config
2 participants