e2e test framework #77

surajssd · 2017-06-27T10:07:27Z

Basic work for e2e test framework

Fixes #59

surajssd · 2017-06-27T10:09:28Z

This won't compile in this directory right now because we have old client-go, right now to test it, just copy this script somewhere outside and following

go get k8s.io/client-go/...
go get -u k8s.io/apimachinery/...
go run main.go

surajssd · 2017-06-27T10:10:07Z

@kadel @containscafeine initial feedback on this is welcome!

surajssd · 2017-06-28T12:53:16Z

add the new examples like single_file
add the new examples like envFrom
move from main to using testing framework this will need a removal of sync.WaitGroup
make the tests run in parallel by using t.Run on each test in table driven test and then add a t.Parallel in each test run.
delete ns even if the test fails
make the example test file name configurable relative to repo name(urgent)
once namespace support to kedge create is added then use that here.

surajssd · 2017-06-28T19:16:23Z

Now since it is using golang test suite so the way to test is to goto e2e folder and run:

go test -v

pradeepto · 2017-06-29T13:03:13Z

Very cool @surajssd . Thanks for taking this up proactively.

concaf · 2017-07-03T06:56:51Z

Great work @surajssd, just that let's make sure this goes in small incremental PRs which are easier to review. I know this is blocked on updating client-go (:sob:), but yeah, this is for whenever that gets sorted.

surajssd · 2017-08-21T05:36:02Z

@cdrage TBH in my workflow I never use make bin.

I just use:

make install

and then run kedge commands from wherever possible.

Now how do we make sure that people should have binary using make bin before running, the end to end tests.

concaf · 2017-08-21T06:19:33Z

@surajssd the wordpress ns was failing for me and @surajnarwade the other day, was that fixed?

surajssd · 2017-08-21T06:21:56Z

@containscafeine I might need to sit with you and get that resolved, because it works for the minikube I have, will have to figure why it fails.

cdrage · 2017-08-21T14:37:34Z

@surajssd In a lot of projects, there is no make install but rather the downloading of a binary from a website and moving to /usr/local/bin

make bin to make the binary and then make test-e2e for the tests. I'd much rather not use make install

concaf · 2017-08-22T06:41:42Z

@surajssd works after the changes you suggested -

diff --git a/examples/wordpress/wordpress.yaml b/examples/wordpress/wordpress.yaml
index 1384034..d9df6f6 100644
--- a/examples/wordpress/wordpress.yaml
+++ b/examples/wordpress/wordpress.yaml
@@ -25,9 +25,12 @@ containers:
       httpGet:
         path: /
         port: 80
+      initialDelaySeconds: 20
+      timeoutSeconds: 5
+

surajssd · 2017-08-22T07:28:45Z

@cdrage

@surajssd In a lot of projects, there is no make install but rather the downloading of a binary from a website and moving to /usr/local/bin make bin to make the binary and then make test-e2e for the tests. I'd much rather not use make install

done changes!

@containscafeine also done changes as you have shown in the diff!

surajnarwade · 2017-08-22T09:32:22Z

Works for me :)

$ make test-e2e 
go test github.com/kedgeproject/kedge/tests/e2e
ok  	github.com/kedgeproject/kedge/tests/e2e	191.008s

surajssd · 2017-08-28T18:58:20Z

Can someone give LGTM and merge this one, I am not sure what is it blocked on now ? @cdrage @tkral @containscafeine

cdrage · 2017-08-28T19:39:02Z

So obviously not an error, but is there any way to add a time-out to these tests? For example, I ran the tests against a "dirty" minikube instance that had the previous namespaces and it took over 100seconds:

github.com/kedgeproject/kedge  pr_77 ✔                                                                                             12d  
▶ make test-e2e 
go test github.com/kedgeproject/kedge/tests/e2e
--- FAIL: Test_Integration (0.04s)
    --- FAIL: Test_Integration/Normal_Wordpress_test (0.02s)
        e2e_test.go:344: error creating namespace: namespaces "wordpress" already exists
    --- FAIL: Test_Integration/Testing_extraResources (0.02s)
        e2e_test.go:344: error creating namespace: namespaces "extraresources" already exists
    --- FAIL: Test_Integration/Testing_health (0.01s)
        e2e_test.go:344: error creating namespace: namespaces "health" already exists
    --- FAIL: Test_Integration/Testing_customVol (0.00s)
        e2e_test.go:344: error creating namespace: namespaces "customvol" already exists
FAIL
FAIL    github.com/kedgeproject/kedge/tests/e2e 105.803s
Makefile:60: recipe for target 'test-e2e' failed
make: *** [test-e2e] Error 1

cdrage · 2017-08-28T19:44:44Z

Another thing as well, I have no indication on what is being created / made... Can you somehow add some debug messages / info messages? make test-e2e doesn't show indication of it doing anything despite it creating containers and namespaces...

Literally all I get is this:

▶ make test-e2e 
go test github.com/kedgeproject/kedge/tests/e2e
ok      github.com/kedgeproject/kedge/tests/e2e 155.270s

Which isn't okay to see what tests have passed or not 👎

github.com/kedgeproject/kedge  pr_77 ✔                                                                                                                                                                                                                                      12d  
▶ kubectl get po
No resources found.

github.com/kedgeproject/kedge  pr_77 ✔                                                                                                                                                                                                                                      12d  
▶ kubectl get ns
NAME           STATUS    AGE
configmap      Active    4s
default        Active    1m
health         Active    4s
healthchecks   Active    4s
kube-public    Active    1m
kube-system    Active    1m
singlefile     Active    4s

github.com/kedgeproject/kedge  pr_77 ✔                                                                                                                                                                                                                                      12d  
▶ kubectl get po --all-namespaces
NAMESPACE      NAME                          READY     STATUS              RESTARTS   AGE
configmap      database-721670202-m62h0      0/1       ContainerCreating   0          8s
configmap      web-2351182281-zjjfl          0/1       ContainerCreating   0          6s
health         database-1377422287-t9rrz     0/1       ContainerCreating   0          8s
health         web-3359867758-snx34          0/1       ContainerCreating   0          7s
healthchecks   database-255470460-rsmbw      0/1       ContainerCreating   0          8s
healthchecks   web-2329707375-9s5pg          0/1       ContainerCreating   0          7s
healthchecks   web-2329707375-s27mr          0/1       ContainerCreating   0          7s
kube-system    kube-addon-manager-minikube   1/1       Running             0          1m
kube-system    kube-dns-1301475494-p6glm     3/3       Running             0          52s
kube-system    kubernetes-dashboard-nj4tg    1/1       Running             0          52s
singlefile     database-4172966911-mfhzm     0/1       ContainerCreating   0          8s
singlefile     wordpress-1717859168-bq3x2    0/1       ContainerCreating   0          7s

Another thing too, is there any way to rate-limit how many go routines are being created / what containers are being created? This slows down my computer A LOT and I wouldn't be surprised if this fails on Semaphore / Fabric8.

I believe we brought up the discussion in regards to adding a parameter to limit the amount of go routines (using go channels) with a passed in variable, for example: make test-e2e PARALLEL=2 or something.

surajssd · 2017-08-31T06:55:07Z

@cdrage thanks for the review

So obviously not an error, but is there any way to add a time-out to these tests? For example, I ran the tests against a "dirty" minikube instance that had the previous namespaces and it took over 100seconds:

It took 100seconds because while some of the tests failed, some other tests were being run which passed. And golang test suite will show output only when tests are completed failed or passed. Since all the tests are run in parallel.

Another thing as well, I have no indication on what is being created / made... Can you somehow add some debug messages / info messages? make test-e2e doesn't show indication of it doing anything despite it creating containers and namespaces...

I have added bunch of log messages already, but the problem is with the golang test suite which will output only when all of tests have completed. So you won't see anything when it is in progress, but only after it has failed or passed.

About verbosity

I did not add verbosity to the tests because it is too much of output. Also when tests fail, the output is verbose anyway for the failed test. Like the following test:

Simulating an errorsome situation

$ kproject.sh wordpress
+ kubectl create ns wordpress
namespace "wordpress" created
+ kubectl config set-context minikube --namespace wordpress
Context "minikube" modified.


$ k get ns
NAME          STATUS    AGE
default       Active    13d
kube-public   Active    13d
kube-system   Active    13d
wordpress     Active    1m


$ make bin
go build -ldflags="-w -X github.com/kedgeproject/kedge/cmd.GITCOMMIT=0266683" -o kedge main.go

$ make test-e2e 
go test github.com/kedgeproject/kedge/tests/e2e
--- FAIL: Test_Integration (0.04s)
    --- FAIL: Test_Integration/Normal_Wordpress_test (0.01s)
        e2e_test.go:344: error creating namespace: namespaces "wordpress" already exists
FAIL
FAIL    github.com/kedgeproject/kedge/tests/e2e 117.438s
Makefile:60: recipe for target 'test-e2e' failed
make: *** [test-e2e] Error 1

Above you can see I see only output of failed test. But when you make it verbose for everything then it becomes hard to parse what failed and what passed due to amount of output that is thrown on the stdout.

Like see following with verbosity added:

$ make test-e2e
go test -v github.com/kedgeproject/kedge/tests/e2e
=== RUN   Test_Integration
=== RUN   Test_Integration/Testing_configMap
=== RUN   Test_Integration/Testing_customVol
=== RUN   Test_Integration/Testing_envFrom
=== RUN   Test_Integration/Testing_extraResources
=== RUN   Test_Integration/Testing_health
=== RUN   Test_Integration/Testing_healthChecks
=== RUN   Test_Integration/Testing_secret
=== RUN   Test_Integration/Testing_single_file
=== RUN   Test_Integration/Normal_Wordpress_test
--- FAIL: Test_Integration (0.03s)
    --- PASS: Test_Integration/Testing_single_file (31.88s)
        e2e_test.go:346: namespace "singlefile" created
        e2e_test.go:354: service "database" created
                deployment "database" created

[SNIP]

        e2e_test.go:183: "wordpress:8080" is running!
        e2e_test.go:370: Successfully pinged all endpoints!
        e2e_test.go:201: successfully deleted namespace: "singlefile"
    --- FAIL: Test_Integration/Normal_Wordpress_test (0.01s)
        e2e_test.go:344: error creating namespace: namespaces "wordpress" already exists
    --- PASS: Test_Integration/Testing_secret (32.17s)

[SNIP 300 lines]

        e2e_test.go:370: Successfully pinged all endpoints!
        e2e_test.go:201: successfully deleted namespace: "health"
FAIL
exit status 1
FAIL    github.com/kedgeproject/kedge/tests/e2e 87.011s
Makefile:60: recipe for target 'test-e2e' failed
make: *** [test-e2e] Error 1

Literally all I get is this:

I have added a verbose flag to the test suite, anyway. So now you will see more output than is there.

I believe we brought up the discussion in regards to adding a parameter to limit the amount of go routines (using go channels) with a passed in variable, for example: make test-e2e PARALLEL=2 or something.

Now you rate limit the number of go processes that run in parallel using the same: make test-e2e PARALLEL=2 otherwise it defaults to the number of processors that are there on your go test running machine. See this stackoverflow question for some background on this one.

surajssd · 2017-08-31T07:48:01Z

@cdrage also added instructions in docs/development.md

cdrage · 2017-08-31T10:50:23Z

tests/e2e/e2e_test.go

+	for _, test := range tests {
+		test := test // capture range variable
+		t.Run(test.TestName, func(t *testing.T) {
+			t.Parallel()


See: https://golang.org/pkg/testing/ with t.Parallel() regardless of what you put as GOMAXPROCS it still runs everything at the same time. Deploying every test to Kubernetes.

Perhaps removing this would help...

Even with me setting the GOMAXPROCS to something lower, I still have every test being deploying at the same time to Kubernetes, which isn't helping.. There should be a limiter to how many tests are ran at a time.

@cdrage I think I know what is confusing you, follow the following steps.

I get this when I run tests:

$ make test-e2e PARALLEL=2 go test -parallel=2 -v github.com/kedgeproject/kedge/tests/e2e === RUN Test_Integration === RUN Test_Integration/Testing_configMap === RUN Test_Integration/Testing_customVol === RUN Test_Integration/Testing_envFrom === RUN Test_Integration/Testing_extraResources === RUN Test_Integration/Testing_health === RUN Test_Integration/Testing_healthChecks === RUN Test_Integration/Testing_secret === RUN Test_Integration/Testing_single_file === RUN Test_Integration/Normal_Wordpress_test ...

Above output shows that all of them are running, but

In another window, while above is happening I could see:

$ kubectl get ns -w NAME STATUS AGE default Active 14d kube-public Active 14d kube-system Active 14d configmap Active 0s singlefile Active 0s configmap Terminating 16s wordpress Active 0s configmap Terminating 38s configmap Terminating 38s wordpress Terminating 33s health Active 0s ...

In above output you can see, configmap and singlefile were created; then configmap was done and wordpress was started and so on. So at anytime only two tests at maximum are running.

Also from docs and help:

$ go test -h test [build/test flags] [packages] [build/test flags & test binary flags] In addition to the build flags, the flags handled by 'go test' itself are: -args ... -parallel n Allow parallel execution of test functions that call t.Parallel. The value of this flag is the maximum number of tests to run simultaneously; by default, it is set to the value of GOMAXPROCS. Note that -parallel only applies within a single test binary. The 'go test' command may run tests for different packages in parallel as well, according to the setting of the -p flag (see 'go help build'). ...

cdrage · 2017-08-31T10:52:28Z

See my above comment, I don't think we should be running this much in parallel... Even on a beefy machine it slows everything down deploying at the same time. GOMAXPROCS does nothing (i've gone ahead and tested it).

surajssd · 2017-08-31T11:06:28Z

@cdrage you can have all tests run serially by running make test-e2e PARALLEL=1. But it can take more time about 7mins.

cdrage · 2017-08-31T14:02:50Z

@surajssd Yup, that'd do it. PARALLEL works. Thanks! I'll do another test.

cdrage · 2017-08-31T15:50:17Z

AWESOME WORK! LGTM!! Works from my end especially with PARALLEL. Now let's get this up on Semaphore / Fabric8. I'll volunteer (since I've done the stuff on Semaphore previously).

cdrage · 2017-08-31T15:50:30Z

@surajssd Merge whenever!

pradeepto · 2017-08-31T15:52:48Z

Great stuff @surajssd, @cdrage and everyone involved in getting this done. Much thanks

Run e2e tests using: ``` make test-e2e ``` The tests run in parallel so you might want to rate limit on number of tests that run in parallel, you can do that by running following: ``` make test-e2e PARALLEL=2 ``` Above will run only two tests in parallel.

surajssd changed the title ~~e2e test framework~~ [WIP] e2e test framework Jun 27, 2017

surajssd mentioned this pull request Jun 27, 2017

Deploy command #72

Merged

concaf added the status/work-in-progress label Jun 27, 2017

surajssd mentioned this pull request Jun 28, 2017

Update client-go to "3.0" #81

Closed

surajssd force-pushed the e2e_tests branch 2 times, most recently from a841c04 to 755e674 Compare June 28, 2017 11:18

surajssd force-pushed the e2e_tests branch from 755e674 to 16ac5c4 Compare June 28, 2017 17:41

surajssd force-pushed the e2e_tests branch from 2b712c1 to bd2e5ab Compare June 30, 2017 06:22

surajssd force-pushed the e2e_tests branch from bd2e5ab to ec925be Compare July 6, 2017 09:51

pradeepto mentioned this pull request Jul 11, 2017

e2e tests - test conversions on running cluster #59

Closed

surajssd force-pushed the e2e_tests branch from ec925be to fc9c0f4 Compare July 13, 2017 11:36

surajssd force-pushed the e2e_tests branch 8 times, most recently from 0c34c2d to d1e96c9 Compare August 1, 2017 13:48

surajssd force-pushed the e2e_tests branch 5 times, most recently from 66a6ebe to 7cde2a9 Compare August 7, 2017 07:25

surajssd force-pushed the e2e_tests branch from 042a8d1 to 11d5da5 Compare August 22, 2017 07:26

surajssd force-pushed the e2e_tests branch from 11d5da5 to 924a9d7 Compare August 31, 2017 06:48

surajssd force-pushed the e2e_tests branch 2 times, most recently from 676ac66 to 5a852f5 Compare August 31, 2017 07:44

cdrage reviewed Aug 31, 2017

View reviewed changes

cdrage approved these changes Aug 31, 2017

View reviewed changes

surajssd added 2 commits September 1, 2017 08:35

Update vendor with needs of running e2e tests

e8648b6

surajssd force-pushed the e2e_tests branch from 5a852f5 to b08e406 Compare September 1, 2017 03:05

surajssd merged commit 93451ea into master Sep 1, 2017

cdrage deleted the e2e_tests branch September 19, 2017 17:18

surajssd mentioned this pull request Oct 23, 2017

fix(e2e tests): make test compile #372

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e test framework #77

e2e test framework #77

surajssd commented Jun 27, 2017 •

edited

Loading

surajssd commented Jun 27, 2017

surajssd commented Jun 27, 2017

surajssd commented Jun 28, 2017 •

edited

Loading

surajssd commented Jun 28, 2017

pradeepto commented Jun 29, 2017

concaf commented Jul 3, 2017

surajssd commented Aug 21, 2017

concaf commented Aug 21, 2017

surajssd commented Aug 21, 2017

cdrage commented Aug 21, 2017

concaf commented Aug 22, 2017

surajssd commented Aug 22, 2017

surajnarwade commented Aug 22, 2017

surajssd commented Aug 28, 2017

cdrage commented Aug 28, 2017

cdrage commented Aug 28, 2017

surajssd commented Aug 31, 2017 •

edited

Loading

surajssd commented Aug 31, 2017

cdrage Aug 31, 2017

surajssd Aug 31, 2017

surajssd Aug 31, 2017

cdrage commented Aug 31, 2017

surajssd commented Aug 31, 2017 •

edited

Loading

cdrage commented Aug 31, 2017

cdrage commented Aug 31, 2017

cdrage commented Aug 31, 2017

pradeepto commented Aug 31, 2017

e2e test framework #77

e2e test framework #77

Conversation

surajssd commented Jun 27, 2017 • edited Loading

surajssd commented Jun 27, 2017

surajssd commented Jun 27, 2017

surajssd commented Jun 28, 2017 • edited Loading

surajssd commented Jun 28, 2017

pradeepto commented Jun 29, 2017

concaf commented Jul 3, 2017

surajssd commented Aug 21, 2017

concaf commented Aug 21, 2017

surajssd commented Aug 21, 2017

cdrage commented Aug 21, 2017

concaf commented Aug 22, 2017

surajssd commented Aug 22, 2017

surajnarwade commented Aug 22, 2017

surajssd commented Aug 28, 2017

cdrage commented Aug 28, 2017

cdrage commented Aug 28, 2017

surajssd commented Aug 31, 2017 • edited Loading

surajssd commented Aug 31, 2017

cdrage Aug 31, 2017

Choose a reason for hiding this comment

surajssd Aug 31, 2017

Choose a reason for hiding this comment

surajssd Aug 31, 2017

Choose a reason for hiding this comment

cdrage commented Aug 31, 2017

surajssd commented Aug 31, 2017 • edited Loading

cdrage commented Aug 31, 2017

cdrage commented Aug 31, 2017

cdrage commented Aug 31, 2017

pradeepto commented Aug 31, 2017

surajssd commented Jun 27, 2017 •

edited

Loading

surajssd commented Jun 28, 2017 •

edited

Loading

surajssd commented Aug 31, 2017 •

edited

Loading

surajssd commented Aug 31, 2017 •

edited

Loading