Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error "Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e//..." is faced when the impl is exactly equals the example #1830

Closed
camilamacedo86 opened this issue Aug 17, 2019 · 14 comments
Closed

Error "Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e//..." is faced when the impl is exactly equals the example #1830

camilamacedo86 opened this issue Aug 17, 2019 · 14 comments

Comments

@camilamacedo86
Copy link
Contributor

camilamacedo86 commented Aug 17, 2019 •
edited
Loading

Bug Report

Bug

What did you do?
Following the e2e test doc guide and impl exactly like the example suggested here

What did you expect to see?
The e2e test work with success.

What did you see instead? Under which circumstances?

~/go/src/github.com/dev4devs-com/postgresql-operator (status_tests_refac) $ make test/e2e
operator-sdk test local ./test/e2e/ --go-test-flags "-v -parallel=2"
INFO[0000] Testing operator locally.                    
=== RUN   TestPostgreSQL
=== RUN   TestPostgreSQL/postgresql-group
=== RUN   TestPostgreSQL/postgresql-group/Cluster
=== PAUSE TestPostgreSQL/postgresql-group/Cluster
=== RUN   TestPostgreSQL/postgresql-group/Cluster2
=== PAUSE TestPostgreSQL/postgresql-group/Cluster2
=== CONT  TestPostgreSQL/postgresql-group/Cluster
=== CONT  TestPostgreSQL/postgresql-group/Cluster2
--- FAIL: TestPostgreSQL (60.74s)
   --- FAIL: TestPostgreSQL/postgresql-group (0.00s)
       --- FAIL: TestPostgreSQL/postgresql-group/Cluster2 (0.55s)
           client.go:57: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster2-1566045200/postgresql-operator) created
           postgresql_test.go:83: failed to initialize cluster resources: clusterroles.rbac.authorization.k8s.io "postgresql-operator" already exists
           client.go:75: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster2-1566045200/postgresql-operator) successfully deleted
       --- FAIL: TestPostgreSQL/postgresql-group/Cluster (60.60s)
           client.go:57: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) created
           client.go:57: resource type ClusterRole with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) created
           client.go:57: resource type ClusterRoleBinding with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) created
           client.go:57: resource type Deployment with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) created
           postgresql_test.go:85: Initialized cluster resources
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           wait_util.go:64: Waiting for full availability of postgresql-operator deployment (0/1)
           postgresql_test.go:95: timed out waiting for the condition
           client.go:75: resource type Deployment with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) successfully deleted
           client.go:75: resource type ClusterRoleBinding with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) successfully deleted
           client.go:75: resource type ClusterRole with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) successfully deleted
           client.go:75: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster-1566045200/postgresql-operator) successfully deleted
FAIL
FAIL	github.com/dev4devs-com/postgresql-operator/test/e2e	61.829s
Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e//...", "-namespacedMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/namespaced-manifest.yaml206846903", "-globalMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/global-manifest.yaml305338282", "-root", "/Users/camilamacedo/go/src/github.com/dev4devs-com/postgresql-operator", "-v", "-parallel=2"}: exit status 1)
Usage:
 operator-sdk test local <path to tests directory> [flags]

Flags:
     --debug                         Enable debug-level logging
     --global-manifest string        Path to manifest for Global resources (e.g. CRD manifests)
     --go-test-flags string          Additional flags to pass to go test
 -h, --help                          help for local
     --image string                  Use a different operator image from the one specified in the namespaced manifest
     --kubeconfig string             Kubeconfig path
     --local-operator-flags string   The flags that the operator needs (while using --up-local). Example: "--flag1 value1 --flag2=value2"
     --molecule-test-flags string    Additional flags to pass to molecule test
     --namespace string              If non-empty, single namespace to run tests in
     --namespaced-manifest string    Path to manifest for per-test, namespaced resources (e.g. RBAC and Operator manifest)
     --no-setup                      Disable test resource creation
     --up-local                      Enable running operator locally with go run instead of as an image in the cluster

Global Flags:
     --verbose   Enable verbose logging

make: *** [test/e2e] Error 1

Please feel free to check its impl and test it. See the code impl here: https://github.com/dev4devs-com/postgresql-operator/tree/master/test/e2e

Environment

  • operator-sdk version:

operator-sdk version: v0.8.0-146-g4ca881a9, commit: 4ca881a (master)

  • go version:

go version go1.12.7 darwin/amd64

  • Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T14:25:20Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-08-16T16:39:41Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

  • Kubernetes cluster kind: OCP

  • Are you writing your operator in ansible, helm, or go? Go

Possible Solution
@camilamacedo86
Copy link
Contributor Author

c/c @AlexNPavel

@camilamacedo86
Copy link
Contributor Author

I already tried operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=2" the error is the same.

@AlexNPavel
Copy link
Contributor

Hello. It looks like one issue you are having is that your resource uses a cluster role. I believe that you currently have that configured via the role.yaml and role_binding.yaml files, which are automatically considered as a namespaced role, which is incorrect in this case. That is why Cluster2 is failing, and that can be resolved by using --go-test-flags "-parallel=1" or --single-namespace, which disables parallel tests (you can also remove the t.Parallel() lines in your test code).

For why the test Cluster did not pass, it is difficult to know without any logs. I believe that if you run the operator locally using --up-local, the test should automatically print out the logs of the operator for you, which can help you debug the issue. Otherwise, you have to use kubectl or oc to look at the operator pod's logs during the test.

@gegles
Copy link

gegles commented Aug 22, 2019

I am facing similar issues... any breakthrough?

@camilamacedo86
Copy link
Contributor Author

camilamacedo86 commented Sep 2, 2019 •
edited
Loading

Hi @AlexNPavel,

In my project https://github.com/dev4devs-com/postgresql-operator if I run operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=1" and by removing the t.Parallel() as you suggested it still not working.

Following the error.

 $ operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=1"
INFO[0000] Testing operator locally.                    
=== RUN   TestPostgreSQL
=== RUN   TestPostgreSQL/postgresql-group
=== RUN   TestPostgreSQL/postgresql-group/Cluster
=== PAUSE TestPostgreSQL/postgresql-group/Cluster
=== RUN   TestPostgreSQL/postgresql-group/Cluster2
=== PAUSE TestPostgreSQL/postgresql-group/Cluster2
=== CONT  TestPostgreSQL/postgresql-group/Cluster
=== CONT  TestPostgreSQL/postgresql-group/Cluster2
--- FAIL: TestPostgreSQL (1.65s)
    --- FAIL: TestPostgreSQL/postgresql-group (0.00s)
        --- FAIL: TestPostgreSQL/postgresql-group/Cluster (0.50s)
            client.go:57: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster-1567438884/postgresql-operator) created
            postgresql_test.go:90: failed to initialize cluster resource: clusterroles.rbac.authorization.k8s.io "postgresql-operator" already exists
            client.go:75: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster-1567438884/postgresql-operator) successfully deleted
        --- FAIL: TestPostgreSQL/postgresql-group/Cluster2 (0.51s)
            client.go:57: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster2-1567438884/postgresql-operator) created
            postgresql_test.go:90: failed to initialize cluster resource: clusterroles.rbac.authorization.k8s.io "postgresql-operator" already exists
            client.go:75: resource type ServiceAccount with namespace/name (postgresql-postgresql-group-cluster2-1567438884/postgresql-operator) successfully deleted
FAIL
FAIL	github.com/dev4devs-com/postgresql-operator/test/e2e	4.357s
Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/namespaced-manifest.yaml579417997", "-globalMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/global-manifest.yaml374207368", "-root", "/Users/camilamacedo/go/src/github.com/dev4devs-com/postgresql-operator", "-v", "-parallel=1"}: exit status 1)

Also, note that I am NOT able to see any error when I run operator-sdk up local and/or when I test the operator project manually by applying it in a cluster.

@camilamacedo86
Copy link
Contributor Author

camilamacedo86 commented Sep 2, 2019 •
edited
Loading

Hi @AlexNPavel,

I could confirm that it is a bug since the example is not working with the operator-sdk version "v0.10.0-28-g8f1d0a04" (master branch) too. Note that the same error is faced. Could you please help us with by flagging it as a bug for the test feature of the framework which needs to be addressed?

camilamacedo@Camilas-MacBook-Pro ~/go/src/github.com/operator-framework/operator-sdk-samples/memcached-operator (master) $ operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=1"
INFO[0000] Testing operator locally.                    
go: creating new go.mod: module github.com/operator-framework/operator-sdk-samples/memcached-operator
go: copying requirements from Gopkg.lock
go: finding github.com/operator-framework/operator-sdk v0.6.0
go: finding k8s.io/helm v2.13.0+incompatible
go: finding golang.org/x/net v0.0.0-20190403144856-b630fd6fe46b
go: finding golang.org/x/tools v0.0.0-20190403183509-8a44e74612bc
go: finding golang.org/x/crypto v0.0.0-20190403202508-8e1b8d32e692
go: downloading github.com/operator-framework/operator-sdk v0.6.0
go: downloading golang.org/x/net v0.0.0-20190403144856-b630fd6fe46b
go: extracting golang.org/x/net v0.0.0-20190403144856-b630fd6fe46b
go: extracting github.com/operator-framework/operator-sdk v0.6.0
go: downloading golang.org/x/tools v0.0.0-20190403183509-8a44e74612bc
go: downloading sigs.k8s.io/controller-tools v0.1.8
go: downloading golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e
go: downloading golang.org/x/crypto v0.0.0-20190403202508-8e1b8d32e692
go: downloading k8s.io/helm v2.13.0+incompatible
go: extracting k8s.io/helm v2.13.0+incompatible
go: extracting golang.org/x/crypto v0.0.0-20190403202508-8e1b8d32e692
go: extracting golang.org/x/sys v0.0.0-20190403152447-81d4e9dc473e
go: extracting golang.org/x/tools v0.0.0-20190403183509-8a44e74612bc
go: extracting sigs.k8s.io/controller-tools v0.1.8
=== RUN   TestMemcached
=== RUN   TestMemcached/memcached-group
=== RUN   TestMemcached/memcached-group/Cluster
=== PAUSE TestMemcached/memcached-group/Cluster
=== RUN   TestMemcached/memcached-group/Cluster2
=== PAUSE TestMemcached/memcached-group/Cluster2
=== CONT  TestMemcached/memcached-group/Cluster
=== CONT  TestMemcached/memcached-group/Cluster2
--- FAIL: TestMemcached (122.83s)
    --- FAIL: TestMemcached/memcached-group (0.00s)
        --- FAIL: TestMemcached/memcached-group/Cluster (60.61s)
            client.go:57: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) created
            client.go:57: resource type Role with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) created
            client.go:57: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) created
            client.go:57: resource type Deployment with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) created
            memcached_test.go:100: Initialized cluster resources
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            memcached_test.go:110: timed out waiting for the condition
            client.go:75: resource type Deployment with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) successfully deleted
            client.go:75: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) successfully deleted
            client.go:75: resource type Role with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) successfully deleted
            client.go:75: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster-1567439532/memcached-operator) successfully deleted
        --- FAIL: TestMemcached/memcached-group/Cluster2 (60.48s)
            client.go:57: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) created
            client.go:57: resource type Role with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) created
            client.go:57: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) created
            client.go:57: resource type Deployment with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) created
            memcached_test.go:100: Initialized cluster resources
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            memcached_test.go:110: timed out waiting for the condition
            client.go:75: resource type Deployment with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) successfully deleted
            client.go:75: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) successfully deleted
            client.go:75: resource type Role with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) successfully deleted
            client.go:75: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster2-1567439593/memcached-operator) successfully deleted
FAIL
FAIL	github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e	123.977s
Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/namespaced-manifest.yaml070769587", "-globalMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/global-manifest.yaml692678006", "-root", "/Users/camilamacedo/go/src/github.com/operator-framework/operator-sdk-samples/memcached-operator", "-v", "-parallel=1"}: exit status 1)
Usage:
  operator-sdk test local <path to tests directory> [flags]

Flags:
      --debug                         Enable debug-level logging
      --global-manifest string        Path to manifest for Global resources (e.g. CRD manifests)
      --go-test-flags string          Additional flags to pass to go test
  -h, --help                          help for local
      --image string                  Use a different operator image from the one specified in the namespaced manifest
      --kubeconfig string             Kubeconfig path
      --local-operator-flags string   The flags that the operator needs (while using --up-local). Example: "--flag1 value1 --flag2=value2"
      --molecule-test-flags string    Additional flags to pass to molecule test
      --namespace string              If non-empty, single namespace to run tests in
      --namespaced-manifest string    Path to manifest for per-test, namespaced resources (e.g. RBAC and Operator manifest)
      --no-setup                      Disable test resource creation
      --up-local                      Enable running operator locally with go run instead of as an image in the cluster

Global Flags:
      --verbose   Enable verbose logging

camilamacedo@Camilas-MacBook-Pro ~/go/src/github.com/operator-framework/operator-sdk-samples/memcached-operator (master) $

@AlexNPavel
Copy link
Contributor

@camilamacedo86 Can you try with the --up-local flag? That flag runs the operator locally on your machine instead of the cluster and also prints out the logs of the operator which can help you debug the issue. This may also fix the problem entirely as the problem you are having is usually due to the operator image not existing in your cluster.

If your issue has to do with the image not existing in the cluster, there are 2 ways to fix it.

  1. Push the image to a docker registry such as quay, and set the correct image name and set the pull policy to Always in the deploy/operator.yaml file.
  2. Build the image in the docker registry of the cluster and set the correct image name and set the pull policy to Never in deploy/operator.yaml. If using minikube, you can use eval $(minikube docker-env) and then build the image as usual.

I believe there is also a way to build the image or push it to the OCP cluster's internal image registry, but I am unfamiliar with OCP specific features, so I am unsure how to do that.

Another important thing to note is that the test will fail if any of the resources already exist in the cluster. The reason that your Postgres test failed is because the cluster role for the operator already existed.

@camilamacedo86
Copy link
Contributor Author

camilamacedo86 commented Sep 3, 2019 •
edited
Loading

HI @AlexNPavel,

Thank you for your reply.

IMPORTANT: note that the example is not using the latest version of the SDK where the metric impl was changed. See here the _, err = metrics.ExposeMetricsPort(ctx, metricsPort) when it would be done as here in my operator

So, shows that the operator-sdk test local ./test/e2e is working for the versions < 9.0 only and that the flag --up-local is no longer working with go modules.

Following the tests performed.

With github.com/operator-framework/operator-sdk-samples/memcached-operator

  • operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=1" - 👍 Worked
  • operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=2" - 👍 Worked
  • operator-sdk test local ./test/e2e - 👍Worked
  • operator-sdk test local ./test/e2e --debug - 👍 Worked
  • operator-sdk test local ./test/e2e --verbose - 👍 Worked
  • operator-sdk test local ./test/e2e --up-local --namespace=memcached - 👎 It is not working.
    Following the error.
$ export OPERATOR_NAME=memcached-operator
$ export WATCH_NAMESPACE=memcached
$ operator-sdk test local ./test/e2e --up-local --namespace=memcached
INFO[0000] Testing operator locally.                    
time="2019-09-03T10:00:20+01:00" level=info msg="Started local operator"
--- FAIL: TestMemcached (121.13s)
    --- FAIL: TestMemcached/memcached-group (0.00s)
        --- FAIL: TestMemcached/memcached-group/Cluster (60.01s)
            memcached_test.go:100: Initialized cluster resources
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            memcached_test.go:110: timed out waiting for the condition
        --- FAIL: TestMemcached/memcached-group/Cluster2 (60.00s)
            memcached_test.go:100: Initialized cluster resources
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            wait_util.go:55: Waiting for availability of memcached-operator deployment
            memcached_test.go:110: timed out waiting for the condition
FAIL
time="2019-09-03T10:02:22+01:00" level=info msg="Local operator stdout: "
time="2019-09-03T10:02:22+01:00" level=info msg="Local operator stderr: {\"level\":\"info\",\"ts\":1567501220.974571,\"logger\":\"cmd\",\"msg\":\"Go Version: go1.12.7\"}\n{\"level\":\"info\",\"ts\":1567501220.9746182,\"logger\":\"cmd\",\"msg\":\"Go OS/Arch: darwin/amd64\"}\n{\"level\":\"info\",\"ts\":1567501220.974626,\"logger\":\"cmd\",\"msg\":\"Version of operator-sdk: v0.6.0\"}\n{\"level\":\"info\",\"ts\":1567501220.982945,\"logger\":\"leader\",\"msg\":\"Trying to become the leader.\"}\n{\"level\":\"info\",\"ts\":1567501220.982997,\"logger\":\"leader\",\"msg\":\"Skipping leader election; not running in a cluster.\"}\n{\"level\":\"info\",\"ts\":1567501221.038113,\"logger\":\"cmd\",\"msg\":\"Registering Components.\"}\n{\"level\":\"info\",\"ts\":1567501221.038236,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"memcached-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567501221.0383239,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"memcached-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567501221.071306,\"logger\":\"metrics\",\"msg\":\"Skipping metrics Service creation; not running in a cluster.\"}\n{\"level\":\"info\",\"ts\":1567501221.07133,\"logger\":\"cmd\",\"msg\":\"Starting the Cmd.\"}\n{\"level\":\"info\",\"ts\":1567501221.171699,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting Controller\",\"controller\":\"memcached-controller\"}\n{\"level\":\"info\",\"ts\":1567501221.273242,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting workers\",\"controller\":\"memcached-controller\",\"worker count\":1}\n"
FAIL	github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e	121.879s
Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/empty.yaml566898283", "-globalMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/global-manifest.yaml178581710", "-root", "/Users/camilamacedo/go/src/github.com/operator-framework/operator-sdk-samples/memcached-operator", "-singleNamespace", "-parallel=1", "-localOperator"}: exit status 1)

With my operator github.com/dev4devs-com/postgresql-operator:

 $  operator-sdk test local ./test/e2e --up-local --namespace=postgresql
INFO[0000] Testing operator locally.                    
time="2019-09-03T10:31:46+01:00" level=info msg="Started local operator"
--- FAIL: TestPostgreSQL (120.27s)
    --- FAIL: TestPostgreSQL/postgresql-group (0.00s)
        --- FAIL: TestPostgreSQL/postgresql-group/Cluster (60.00s)
            postgresql_test.go:92: Initialized cluster resource
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            postgresql_test.go:102: timed out waiting for the condition
        --- FAIL: TestPostgreSQL/postgresql-group/Cluster2 (60.01s)
            postgresql_test.go:92: Initialized cluster resource
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            wait_util.go:55: Waiting for availability of postgresql-operator deployment
            postgresql_test.go:102: timed out waiting for the condition
FAIL
time="2019-09-03T10:33:46+01:00" level=info msg="Local operator stdout: "
time="2019-09-03T10:33:46+01:00" level=info msg="Local operator stderr: {\"level\":\"info\",\"ts\":1567503106.64674,\"logger\":\"cmd\",\"msg\":\"Go Version: go1.12.7\"}\n{\"level\":\"info\",\"ts\":1567503106.6469572,\"logger\":\"cmd\",\"msg\":\"Go OS/Arch: darwin/amd64\"}\n{\"level\":\"info\",\"ts\":1567503106.646966,\"logger\":\"cmd\",\"msg\":\"Version of operator-sdk: v0.10.0+git\"}\n{\"level\":\"info\",\"ts\":1567503106.650644,\"logger\":\"leader\",\"msg\":\"Trying to become the leader.\"}\n{\"level\":\"info\",\"ts\":1567503106.65066,\"logger\":\"leader\",\"msg\":\"Skipping leader election; not running in a cluster.\"}\n{\"level\":\"info\",\"ts\":1567503106.723727,\"logger\":\"cmd\",\"msg\":\"Registering Components.\"}\n{\"level\":\"info\",\"ts\":1567503106.7243378,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"backup-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.725725,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"backup-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.725842,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"backup-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.725895,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"backup-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.725976,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"postgresql-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.726018,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"postgresql-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.72606,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"postgresql-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.72612,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting EventSource\",\"controller\":\"postgresql-controller\",\"source\":\"kind source: /, Kind=\"}\n{\"level\":\"info\",\"ts\":1567503106.72621,\"logger\":\"cmd\",\"msg\":\"Could not generate and serve custom resource metrics\",\"error\":\"operator run mode forced to local\"}\n{\"level\":\"info\",\"ts\":1567503106.767759,\"logger\":\"metrics\",\"msg\":\"Skipping metrics Service creation; not running in a cluster.\"}\n{\"level\":\"info\",\"ts\":1567503106.7819352,\"logger\":\"cmd\",\"msg\":\"Could not create ServiceMonitor object\",\"error\":\"no ServiceMonitor registered with the API\"}\n{\"level\":\"info\",\"ts\":1567503106.781954,\"logger\":\"cmd\",\"msg\":\"Install prometheus-operator in your cluster to create ServiceMonitor objects\",\"error\":\"no ServiceMonitor registered with the API\"}\n{\"level\":\"info\",\"ts\":1567503106.7819579,\"logger\":\"cmd\",\"msg\":\"Starting the Cmd.\"}\n{\"level\":\"info\",\"ts\":1567503106.883023,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting Controller\",\"controller\":\"postgresql-controller\"}\n{\"level\":\"info\",\"ts\":1567503106.883022,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting Controller\",\"controller\":\"backup-controller\"}\n{\"level\":\"info\",\"ts\":1567503106.98737,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting workers\",\"controller\":\"postgresql-controller\",\"worker count\":1}\n{\"level\":\"info\",\"ts\":1567503106.9873762,\"logger\":\"kubebuilder.controller\",\"msg\":\"Starting workers\",\"controller\":\"backup-controller\",\"worker count\":1}\n"
FAIL	github.com/dev4devs-com/postgresql-operator/test/e2e	121.529s
Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/empty.yaml573345610", "-globalMan", "/var/folders/r_/ym6yn3cd0q3fm5jcgcwvj4z00000gn/T/global-manifest.yaml461284897", "-root", "/Users/camilamacedo/go/src/github.com/dev4devs-com/postgresql-operator", "-singleNamespace", "-parallel=1", "-localOperator"}: exit status 1)
Usage:
  operator-sdk test local <path to tests directory> [flags]

IMHO, following the points that we need to address.

Docs

  • Specifying that we need permission to apply the files. We need to add something as follows.

To test your operator you need be logged in as a user with cluster privileges like the system:admin user. E.g. By using: oc login -u system:admin.

  • Add the info with an example to use the flag to run locally operator-sdk test local ./test/e2e --up-local --namespace=memcached and that for it is required export the envars export OPERATOR_NAME=memcached-operator and export WATCH_NAMESPACE=memcached.

  • Make the user beware and remember the image specified needs to be accessible (public published n the registry)

Code

  • Upgrade the example to use the operator-sdk >= 10
  • Ensure that the e2e tests will be working with the example by using the flag --up-local and not.

Hi @AlexNPavel, WDYT? Is it make sense?

@AlexNPavel
Copy link
Contributor

@camilamacedo86 It looks like the reason that the --up-local test did not work for the sample is because the tests were written before the release of v0.2.0 of the SDK and weren't updated since. In v0.2.0, we added the WaitForOperatorDeployment function, which is required for --up-local to work correctly: #781

We mention the use of WaitForOperatorDeployment in doc/test-framework/writing-e2e-tests, but it looks like the samples still use WaitForDeployment: https://github.com/operator-framework/operator-sdk-samples/blob/master/memcached-operator/test/e2e/memcached_test.go#L108

I will make a PR to update the sample to use the correct funciton and verify that --up-local works.

@AlexNPavel AlexNPavel mentioned this issue Sep 3, 2019
Merged
@camilamacedo86
Copy link
Contributor Author

camilamacedo86 commented Sep 3, 2019 •
edited
Loading

Hi @AlexNPavel,

Really tks for the help and the fast reply.

Note that the example is not using the latest version of the SKD. See here the _, err = metrics.ExposeMetricsPort(ctx, metricsPort) which is deprecated in 0.10 For the versions >= 0.9 it should be done as here in my example operator.

Also, in its logs we can check that it using the version 0.6.0 \"Version of operator-sdk: v0.6.0\"} which may explain why @gegles and I have been facing issues in order to use it with projects built with the latest version.

@AlexNPavel
Copy link
Contributor

@camilamacedo86 There is an open issue for that in the samples repo, but nobody has been working on it it seems: operator-framework/operator-sdk-samples#68. We need to get somebody to work on that.

@camilamacedo86
Copy link
Contributor Author

HI @AlexNPavel,
I will do the operator-framework/operator-sdk-samples#68.
Now (after the pr be merged), the e2e test is working locally in my operator but not in the cluster.
I will do more tests then I will open an issue more specific in order to make easier we address it.

@camilamacedo86 camilamacedo86 mentioned this issue Sep 5, 2019
Merged
@pratikjagrut
Copy link
Contributor

pratikjagrut commented Oct 3, 2019 •
edited
Loading

@camilamacedo86 This issue is closed but I'm still facing a similar problem in running tests(in my project and in sample memcached-operator too).

Here is my output of operator-sdk test local ./test/e2e --go-test-flags "-v -parallel=1"

INFO[0000] Testing operator locally.                    
=== RUN   TestMemcached
=== RUN   TestMemcached/memcached-group
=== RUN   TestMemcached/memcached-group/Cluster
=== PAUSE TestMemcached/memcached-group/Cluster
=== RUN   TestMemcached/memcached-group/Cluster2
=== PAUSE TestMemcached/memcached-group/Cluster2
=== CONT  TestMemcached/memcached-group/Cluster
=== CONT  TestMemcached/memcached-group/Cluster2
--- FAIL: TestMemcached (120.54s)
    --- FAIL: TestMemcached/memcached-group (0.00s)
        --- FAIL: TestMemcached/memcached-group/Cluster (60.09s)
            client.go:57: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) created
            client.go:57: resource type Role with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) created
            client.go:57: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) created
            client.go:57: resource type Deployment with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) created
            memcached_test.go:100: Initialized cluster resources
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            memcached_test.go:110: timed out waiting for the condition
            client.go:75: resource type Deployment with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) successfully deleted
            client.go:75: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) successfully deleted
            client.go:75: resource type Role with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) successfully deleted
            client.go:75: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster-1570109205/memcached-operator) successfully deleted
        --- FAIL: TestMemcached/memcached-group/Cluster2 (60.42s)
            client.go:57: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) created
            client.go:57: resource type Role with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) created
            client.go:57: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) created
            client.go:57: resource type Deployment with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) created
            memcached_test.go:100: Initialized cluster resources
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            wait_util.go:64: Waiting for full availability of memcached-operator deployment (0/1)
            memcached_test.go:110: timed out waiting for the condition
            client.go:75: resource type Deployment with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) successfully deleted
            client.go:75: resource type RoleBinding with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) successfully deleted
            client.go:75: resource type Role with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) successfully deleted
            client.go:75: resource type ServiceAccount with namespace/name (memcached-memcached-group-cluster2-1570109265/memcached-operator) successfully deleted
FAIL
FAIL	github.com/operator-framework/operator-sdk-samples/memcached-operator/test/e2e	120.598s
Error: failed to build test binary: (failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/tmp/namespaced-manifest.yaml154514916", "-globalMan", "/tmp/global-manifest.yaml470907891", "-root", "/home/pjagrut/go/src/github.com/operator-framework/operator-sdk-samples/memcached-operator", "-v", "-parallel=1"}: exit status 1)
Usage:
  operator-sdk test local <path to tests directory> [flags]

Flags:
      --debug                         Enable debug-level logging
      --global-manifest string        Path to manifest for Global resources (e.g. CRD manifests)
      --go-test-flags string          Additional flags to pass to go test
  -h, --help                          help for local
      --image string                  Use a different operator image from the one specified in the namespaced manifest
      --kubeconfig string             Kubeconfig path
      --local-operator-flags string   The flags that the operator needs (while using --up-local). Example: "--flag1 value1 --flag2=value2"
      --molecule-test-flags string    Additional flags to pass to molecule test
      --namespace string              If non-empty, single namespace to run tests in
      --namespaced-manifest string    Path to manifest for per-test, namespaced resources (e.g. RBAC and Operator manifest)
      --no-setup                      Disable test resource creation
      --up-local                      Enable running operator locally with go run instead of as an image in the cluster

Global Flags:
      --verbose   Enable verbose logging

operator-sdk version

operator-sdk version: "v0.10.0-70-g1a5772eb", commit: "1a5772eb71859693c9c0fa5042d33599df30b29f", go version: "go1.12.9 linux/amd64"

@camilamacedo86
Copy link
Contributor Author

Could you raise a new issue @pratikjagrut for we are able to analyse it specific scenario?

@pratikjagrut pratikjagrut mentioned this issue Oct 4, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@AlexNPavel @gegles @camilamacedo86 @pratikjagrut