Unit test fail randomly

[laptua]

Unit test fail randomly

test data contains array of only one element. Test case override to add more elements. Repeatedly run the same test without change. Testcase occasionally failed. If test data contains multiple elements, i.e. without override with set command in testcase. Problem do not exist.

Testdata

    ingress:
      enable: false
      host: test.com
      path: /test
      port: web
    extensions:
      ingress:
          - paths:
              - /api/building-calculator/health
            port: web
            matchDirective: "PathPrefixStrip"
            rewrite: /actuator/health

Testcase

suite: Test Ingress extensions 2 or more Paths without rewrite
templates:
  - deploymentSpec/extensions/ingress.yaml
values:
  - ./values/ingress_ex_default.yaml
tests:
  - it: ingress documents (one for ingress, one for middleware)
    asserts:
      - hasDocuments:
          count: 4
  - it: ingress kind and path
    set:
      deploymentSpec.app.extensions.ingress[0].matchDirective: PathPrefixStrip
      deploymentSpec.app.extensions.ingress[0].paths:
        - /welcome/api/wiremock
        - /loyalty/api/wiremock
    documentIndex: 0
    asserts:
      - equal:
          path: metadata.annotations.kubernetes\.io/ingress\.class
          value: traefik
      - isKind:
          of: Ingress
      - equal:
          path: spec.rules[0].http.paths[0].path
          value: /welcome/api/wiremock
      - isNull:
          path: spec.rules[0].host
      - equal:
          path: spec.rules[0].http.paths[0].backend.servicePort
          value: 80
      - equal:
          path: spec.rules[0].http.paths[0].backend.serviceName
          value: base-helm-chart-app-svc

Repeat run the same test without change, testcase fail randomly

 PASS  Test Ingress Chain	tests/app-ingress_chain_test.yaml
 PASS  Test Ingress default matchDirective value to PathPrefixStrip	tests/app-ingress_default_test.yaml
 PASS  Test Ingress PathPrefix	tests/app-ingress_pathprefix_test.yaml
 PASS  Test Ingress PathPrefixStrip	tests/app-ingress_pathprefixstrip_test.yaml
 PASS  Test Ingress ReplacePathRegex	tests/app-ingress_replacepathregex_test.yaml
 PASS  Test Ingress extensions 2 or more Paths without rewrite	tests/ingress_ex_default2_test.yaml
 PASS  Test Ingress extensions 1 path with rewrite which force matchDirective to PathPrefix	tests/ingress_ex_default_test.yaml

Charts:      1 passed, 1 total
Test Suites: 7 passed, 7 total
Tests:       29 passed, 29 total
Snapshot:    0 passed, 0 total
Time:        51.38539ms

 PASS  Test Ingress Chain	tests/app-ingress_chain_test.yaml
 PASS  Test Ingress default matchDirective value to PathPrefixStrip	tests/app-ingress_default_test.yaml
 PASS  Test Ingress PathPrefix	tests/app-ingress_pathprefix_test.yaml
 PASS  Test Ingress PathPrefixStrip	tests/app-ingress_pathprefixstrip_test.yaml
 PASS  Test Ingress ReplacePathRegex	tests/app-ingress_replacepathregex_test.yaml
 FAIL  Test Ingress extensions 2 or more Paths without rewrite	tests/ingress_ex_default2_test.yaml
	- ingress extension middleware

		- asserts[3] `equal` fail
			Template:	base-helm-chart/templates/deploymentSpec/extensions/ingress.yaml
			DocumentIndex:	0
			Path:	spec.stripPrefix.prefixes[0]
			Expected to equal:
				/welcome/api/wiremock
			Actual:
				null
			Diff:
				--- Expected
				+++ Actual
				@@ -1,2 +1,2 @@
				-/welcome/api/wiremock
				+null

		- asserts[4] `equal` fail
			Template:	base-helm-chart/templates/deploymentSpec/extensions/ingress.yaml
			DocumentIndex:	0
			Error:
				[1] :
				- null

 PASS  Test Ingress extensions 1 path with rewrite which force matchDirective to PathPrefix	tests/ingress_ex_default_test.yaml

Charts:      1 failed, 0 passed, 1 total
Test Suites: 1 failed, 6 passed, 7 total
Tests:       1 failed, 28 passed, 29 total
Snapshot:    0 passed, 0 total
Time:        52.417587ms

[d3adb5]

I get the same kind of behavior, but with one particular test that uses contains:

 FAIL  Keycloak StatefulSet     keycloak1/tests/statefulset_test.yaml
        - init container downloads OJDBC driver from archive when DB vendor is oracle

                - asserts[0] `contains` fail
                        Template:       keycloak1/charts/keycloak/templates/statefulset.yaml
                        DocumentIndex:  0
                        Error:
                                expect 'spec.template.spec.initContainers' to be an array, got:
                                null

This works fine most of the time, but sometimes I get the above error.

[d3adb5]

Also randomly I get the following Go error:

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/lrills/helm-unittest/pkg/unittest.(*Assertion).Assert(0xc00055a6c0, 0x20?, {0x17bdf40?, 0xc0003dae40}, 0x1, {0x0?, 0x0}, 0xc0006f8be0)
        /root/project/pkg/unittest/assertion.go:53 +0x796
github.com/lrills/helm-unittest/pkg/unittest.(*TestJob).runAssertions(0xc0000c0500, 0xc000ae88c0?, {0x17bdf40, 0xc0003dae40}, 0x8?, {0x0, 0x0}, 0x0)
        /root/project/pkg/unittest/test_job.go:642 +0x139
github.com/lrills/helm-unittest/pkg/unittest.(*TestJob).RunV3(0xc0000c0500, 0xc000ae88c0, 0xc000132e40, 0xc0?, 0xc0006f9400)
        /root/project/pkg/unittest/test_job.go:253 +0x237
github.com/lrills/helm-unittest/pkg/unittest.(*TestSuite).runV3TestJobs(0xc0000dea00, {0x7ffc13386367, 0x7}, 0x40d627?, 0x0)
        /root/project/pkg/unittest/test_suite.go:231 +0x145
github.com/lrills/helm-unittest/pkg/unittest.(*TestSuite).RunV3(0xc0000dea00, {0x7ffc13386367, 0x7}, 0x17bece0?, 0x18?, 0xc0003476c0)
        /root/project/pkg/unittest/test_suite.go:109 +0xad
github.com/lrills/helm-unittest/pkg/unittest.(*TestRunner).runV3SuitesOfChart(0xc000a8fbb8, {0xc0000e2340, 0x7, 0x7ffc13386367?}, {0x7ffc13386367, 0x7})
        /root/project/pkg/unittest/test_runner.go:301 +0x185
github.com/lrills/helm-unittest/pkg/unittest.(*TestRunner).RunV3(0xc000a8fbb8, {0xc0001a61c0, 0xd, 0x0?})
        /root/project/pkg/unittest/test_runner.go:169 +0x1f7
main.glob..func1(0x21c5740?, {0xc0001a61c0, 0xd, 0xe})
        /root/project/cmd/helm-unittest/helm_unittest.go:89 +0x21e
github.com/spf13/cobra.(*Command).execute(0x21c5740, {0xc00003a100, 0xe, 0xe})
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0x21c5740)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:918
main.main()
        /root/project/cmd/helm-unittest/helm_unittest.go:100 +0x25
Error: plugin "unittest" exited with error