WIP: add lifecycle for test

[liangchenye]

Signed-off-by: liangchenye [email protected]

[liangchenye]

I add a lifecycle lib, it helps to test the runtime lifecycle.
It also helps to write test cases with different compositions of 'create/state/start/delete/kill'.

It is the first step to write a test frame for oci.
#66

[wking]

The sharness analog to this file is:

test_expect_success 'Demo lifecycle' "
  subreaper-wrapper ${RUNTIME} create --bundle bundle 25 &
  while test ! -e 25-created; do sleep 1; done &&  # this would be easier with runtime-spec#508, but works as a stop-gap with subreaper-wrapper touching this file when 'create' exits
  ${RUNTIME} state 25 &&
  ${RUNTIME} start 25 &&
  ${RUNTIME} state 25 &&
  ${RUNTIME} delete 25 &&
  ${RUNTIME} state 25 &&  # this line will probably fail.  I'm not sure if/how you handle that in your Go demo
  wait
"

I think that's more compact/obvious/flexible, although I haven't implemented subreaper-wrapper (although this PR does not setup subreaper handling either).

[liangchenye]

In the main function, the lifecycle lib is not that meanful.

I added a new demo pr. I hope it helps to make a clear and 'easy to extend' test frame.
Each bundle + ActionCase is a test case.
We can provide more bundles/actionCases to test a runtime.

[wking]

The sharness analog to this is:

test_expect_success 'create fails when bundle does not exist' "
  test_expect_code 1 ${RUNTIME} create --bundle noexist_bundle 1
"

test_expect_success 'create/kill/destroy succeeds when bundle exists' "
  subreaper-wrapper ${RUNTIME} create --bundle exist_bundle 1 &
  while test ! -e 1-created; do sleep 1; done &&  # this would be easier with runtime-spec#508, but works as a stop-gap with subreaper-wrapper touching this file when 'create' exits
  ${RUNTIME} kill 1 &&
  wait &&
  ${RUNTIME} delete 1 &&
"

test_expect_success 'second create call with same container ID fails' "
  subreaper-wrapper ${RUNTIME} create --bundle exist_bundle 1 &
  while test ! -e 1-created; do sleep 1; done &&  # this would be easier with runtime-spec#508, but works as a stop-gap with subreaper-wrapper touching this file when 'create' exits
  test_expect_code 1 ${RUNTIME} create --bundle exist_bundle 1
  ${RUNTIME} kill 1 &&
  wait &&
  ${RUNTIME} delete 1 &&
"

test_expect_success 'exist_bundle_run_err fails' "
  subreaper-wrapper ${RUNTIME} create --bundle exist_bundle_run_err 1 &
  while test ! -e 1-created; do sleep 1; done &&  # this would be easier with runtime-spec#508, but works as a stop-gap with subreaper-wrapper touching this file when 'create' exits
  test_expect_code 1 wait &&
  ${RUNTIME} delete 1 &&
"

That's a bit more complicated than your cases literal here, but:

• There's no need to write code to iterate over the cases,
• There's no need for an analog to your Lifecycle type,
• You aren't killing successful creates before deleting them, and
• You don't do any of the subreaper magic that you need if you want to see when/how the container process exits.

And I'm cutting corners by assuming that failing commands will exit 1 (opencontainers/runtime-spec#513 only requires non-zero). To do this right in sharness, I'd want to define a test_expect_code_not helper.

[mrunalp]

@liangchenye @wking WDYT about using ginkgo if we want to keep this in go?

[liangchenye]

I think @wking prefers scripts which is easy to modify and run.

To me, ginkgo is good to test a runtime . (And it still needs to have this kind of lifecycle lib)
My plan is we can use 'generate' to prepare the test bundle and 'lifecycle' to run the test.

[wking]

On Tue, Jan 03, 2017 at 11:39:25AM -0800, 梁辰晔 (Liang Chenye) wrote: I think @wking prefers scripts which is easy to modify and run.

Yeah. I don't see a benefit to using Go, since: * I don't expect a lot of programmatic complexity in the test suite itself, so we won't need the features which Go makes more convenient than shell. * Shell is a convenient language for executing binaries and collecting their standard streams and exit codes. But I'm not a maintainer. If the maintainer consensus is that you'd like to use a Go harness for the test suite, then ginkgo looks fine. I'm partial to TAP, and tap-go works, but isn't particularly sophisticated. Although again, I don't see a need for much programmatic complexity in the test harness/suite. [1]: https://github.com/mndrix/tap-go

[mrunalp]

@liangchenye @wking I am fine with using scripts.

[mrunalp]

Discussed with @crosbymichael today. We decided to use ginkgo for the test suite. I will work on getting in the initial code for that.

[liangchenye]

@mrunalp OK, I'll close this PR and implement this kind of test under the ginkgo framework.