Indicate Yes, No, AND not-implemented

[RobDolinMS]

So that people running tests have a clear indication of not just YES/Pass or NO/Fail, we may want to also include a "not implemented" value if a particular functionality is simply not implemented.

(This issue was raised in an OCI Runtime ConCall the week of 5/8 - 5/12/2017)

[wking]

On Wed, May 10, 2017 at 05:55:50PM -0700, Rob Dolin (MSFT) wrote: … if a particular functionality is simply not implemented.

Is that “not implemented in the compliance test suite”? “Not implemented by the runtime” seems like it should be a fail. For runtime-tools, opencontainers/runtime-tools#66 is something like a list of untested spec requirements, although I'm not sure how current it is. I think a more sustainable approach will be filing issues (or PRs :) for any untested conditions. If/when opencontainers/runtime-tools#354 lands and propagates through the test suite, matching tests to runtime requirements will become more straightforward.

[stephenrwalli]

Tests should exist for work that is described as "MAY" in the specifications. If an implementation doesn't implement something that is testable is still testable, a better test case outcome would be Not Implemented rather than Fail.

[wking]

On Mon, May 15, 2017 at 03:59:32PM -0700, Stephen Walli wrote: Tests should exist for work that is described as "MAY" in the specifications.

Do we have anything that fits this? For example, we currently have [1]: Unless support for a valid value is explicitly required, runtimes MAY choose which subset of the valid values it will support. (descended from [2]). How do you turn that into a compliance test?

If an implementation doesn't implement something that is testable is still testable, a better test case outcome would be Not Implemented rather than Fail.

I'm having trouble parsing “is testable is still testable”. It would be nice to have pretty “Tests Not Implemented by runtime-tools” errors where the compliance suite is lacking and “Feature Not Implemented by Runtime” errors where the runtime implementation was lacking an optional feature. But “Tests Not Implemented by runtime-tools” is going to be a heavy lift for runtime-tools until opencontainers/runtime-tools#354 lands; it seems better at the moment to focus on coverage. For “Feature Not Implemented by Runtime” and excepting the valid-values wiggle room granted in the spec I quoted above, I'm not aware of any optional features. For caller sanity, I'd recommend any such features get their own entry in “platforms” [3] with their own compliance track [4] (which is why I'd argued for “protocols” over “platforms”, opencontainers/runtime-spec#570). But maybe “Feature Not Implemented by Runtime” caveats are effectively the same thing (as long as they get distributed as part of the “OCI Certified” badge). [1]: https://github.com/opencontainers/runtime-spec/blame/4aed614c791f4fe291baa2c7cc3dde8734be0876/config.md#L449 [2]: https://github.com/opencontainers/runtime-spec/pull/673/files#diff-c9c91c29b41257aea3a3403cc606ad99R295 [3]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc5/spec.md#platforms [4]: https://github.com/opencontainers/runtime-spec/blob/v1.0.0-rc5/spec.md#notational-conventions

[caniszczyk]

After today's OCI CertWG call, @RobDolinMS is comfortable closing this along with the group