combinational image

[xiekeyang]

Hi all,

The purpose is to talk about improving the agile way of publishing image with cooperation between different departments.

For example, when some image, with many layer parts like applications, jvm, os, want to be published. Following current rules, it needs to generate and publish series of image, as:

• OS (base image)

• JVM (from OS)

• APP (from JVM, from OS)

This is from some consumer requirement. I think image-spec might add some new definition to support this kind of requirement, and bring about some benefits (as described below).
Publishing image might be separated by 2 steps:

1. Upload only JVM and APP as layered-image (means can’t generate rootfs). They need be combined with OS, to generate rootfs and run successfully. The images, in repository, looks like normal image, have also name and reference.

2. Upload combinational image manifest, in which involving above manifest list and their dependency, like below:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.list.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.combinational.image.config.v1+json",
    "size": 7023,
    "digest": "alg:hash"
  },  
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 7143,
      "digest": "alg:hash"
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "size": 7682,
      "digest": "alg:hash"
    }
  ],
}

This is a metadata of whole image. We define a name and reference for it, and publish it on repository. Then consumer can pull it with all layered-image.

I think this way has some benefit:

• Make different branch focus on, and publish only their partly development, needn’t frequently care other branch change (to a certain extent).

• Makes publish and distribution more freely. The final publish job may be taken by application department, or specific department.

• Sometimes departments want to publish their partly image on different Hubs. If we defined the layered-images URL (hashed) in combinational manifest, we will be able to get the whole image from different hubs.

• For image manifest is hashed in combinational manifest, it will not be changed with any layered-image, unless it is re-published.

Here I’d like to discuss with folks about if it make sense.

[wking]

On Thu, Dec 22, 2016 at 02:54:15AM -0800, xiekeyang wrote: 2. Upload combinational image manifest…

Something like this idea has been kicked around before in #102 and #190. @stevvooe pointed out there [1], and I agree, that you can support this use case by manipulating the ‘layers’ entry in the manifest (and making the knock-on updates to ‘rootfs.diff_ids’ in the config). For example, say your OS image has layers A and B. You work up the JVM on top of that, and put the JVM in layer C (so an OS + JVM image would have layers A, B, and C). And you work up your app on top of that, and put the app in layer D (so and OS + JVM + app would have layers A, B, C, and D). If you are careful about forming your higher layers, they'll be fairly independent of the lower layers. So when the OS cuts a new release (say, patching libssl) with layer B', you can publish a new JVM image with A, B', and C and a new app image with A, B', C, and D. That's not always going to work (sometimes you'll have to update your higher layers to get A, B', C, D' or similar), but in many cases your app won't care about the underlying bump.

* Makes publish and distribution more freely. The final publish job may be taken by application department, or specific department.

This is true with the current setup. You can already have the OS team providing layers A and B (and their successors), the JVM team providing layer C (and its successors), and the app team just provide layer D (and its sucecssors as well as a config template). Then the integration / publication team can assemble images from those layers, fill in any missing config sections, and publish the signed image. I sketched out an approach to auth/validation for a workflow like this in [2], although the OCI doesn't have plans to land something like signed name-assertions in the near future (but #400 is still open). Both refs as they stand and the likely future index format from #438 will support publishing and discovering mutable references to these layers. Using the commands from the in-flight opencontainers/image-tools#5 as an example: $ DIGEST=$(oci-cas put my-image-layout <my-app.tar.gz) $ SIZE=$(wc -c my-app.tar.gz | cut -f 1 -d ' ') $ DESCRIPTOR="$(echo "{\"digest\": \"${DIGEST}\", \"size\": ${SIZE}, \"mediaType\": \"application/vnd.oci.image.layer.v1.tar+gzip\"}")" $ echo "${DESCRIPTOR}" | oci-refs put my-image-layout my-app-layer-v1.0

* Sometimes departments want to publish their partly image on different Hubs. If we defined the layered-images URL (hashed) in combinational manifest, we will be able to get the whole image from different hubs.

All of these blobs are content addressable. It doesn't matter which CAS engine the blob is on (that's not part of the content). Clients can attempt to fetch a given blob from any CAS engines they know about. And when you want to support additional engine discovery for a given blob, you can populate the descriptor's ‘urls’ (although you can only touch that at descriptor-creation time). But this seems orthogonal to image-layering. [1]: #190 (comment) [2]: https://github.com/wking/image-spec/blob/3aae6814425df16c13ba89315d7620fce9b6f7b8/name-assertion.md#layers

[xiekeyang]

@wking

I might mostly understand after reading your comment. Maybe we should talk the detailed design principle later. Also I'd like to re-emphasize that:

Purpose of this issue might be some different to #102 and #109. This want to let teams publish their effort as image. Some effort might be just layer (as image patch), but it should (to be discussed) be published as image (with layer, manifest and config).

This is because we can sign the image just on manifest, and consumer can fetch it as normal image fetching way. This kind of patched-image can't be run as oci container individually. But I think it doesn't matter.

The combinational image, which need patch and base image, just set up a manifest list, to combine the manifests and a config in which show how to organize them to run a container (shown the example above).

The combinational image will be publish just after uploading its manifest list, which can be signed.

I prefer this way rather than below:

"layers": [
    {
        "digest": "sha256:abc…"
        "mediaType": "application/vnd.oci.image.manifest.v1+json"
        …
    },
    {
        "digest": "sha256:def…"
        "mediaType": "application/vnd.oci.image.layer.tar+gzip",
        …
    }
]

Please take a comprehensive consideration on my suggestion, thanks a lot!

cc @vbatts @philips @stevvooe

[stevvooe]

This has been suggested several times before and I'm getting a little exhausted explaining why this is a bad idea to do at the image layer but basically, this is the summary:

Composition should be a runtime concern. PODs are one example of this style of composition but there are others. This composition can even be shipped as a separate unit, but that might not necessarily be an OCI image.

[xiekeyang]

@stevvooe thanks for your explain. I read the previous issues #102, #190, involving your comments inside.

Here I won't say my idea (composition image) is prefect, because it is against to immutable building. But we had better to consider the customers requirement. From previous issues, I think @vbatts and @philips might receive similar requirement.

Sometime developers just want to publish their own partly application as (widely using) image, not to care the base image.

And customers want OCI hub to support deployable image directly, as a total solution, during building and publishing stage. They don't want to fix it during deploying and running stage. They think this way is more agile and easy using.

Even we ignore combinational image, how do we look this kind of requirement? Or if this is really bad to OCI ecosystem? Actually I hope we can present a good solution to customers.

Thanks a lot!

[stevvooe]

@xiekeyang This is a great example of the unending scope creep that happens with software engineering. The format already supports the model for which you are asking. There is no need to add more half-baked bells and whistles to meet this use case.

[wking]

Cross-referencing #424, which has some discussion of encoding lineage using existing fields. And for folks who address these use cases by shuffling layers [1], the coming #501 will give you a convenient place to store layer provenance for folks who want to do so. [1]: #508 (comment)

[xiekeyang]

I think #438 is including this requirement. Closed.