Issue when running scorecard on the cert-utils-operator deployed with OLM

[dirgim]

Bug Report

What did you do?
Running the following command on the operator metadata for the cert-utils-operator community operator in this pull request :

operator-sdk scorecard --olm-deployed --csv-path /tmp/operator-test/operator-files/0.0.1/cert-utils-operator.v0.0.1.clusterserviceversion.yaml --namespace test-operator --proxy-image quay.io/operator-framework/scorecard-proxy:master --crds-dir /tmp/operator-test/operator-files/0.0.1 --init-timeout 60

What did you expect to see?
Standard human-readable output of the scorecard command.

What did you see instead? Under which circumstances?
The stderr we got was this:

time="2019-08-05T15:19:09+02:00" level=warning msg="Could not load config file; using flags"
    panic: runtime error: index out of range
     
    goroutine 1 [running]:
    github.com/operator-framework/operator-sdk/internal/pkg/scorecard.runTests(0x0, 0x0, 0x0, 0x0, 0x0)
            src/github.com/operator-framework/operator-sdk/internal/pkg/scorecard/scorecard.go:182 +0x47c8
    github.com/operator-framework/operator-sdk/internal/pkg/scorecard.ScorecardTests(0xc00061ea00, 0xc0009bf080, 0x0, 0xb, 0x0, 0x0)
            src/github.com/operator-framework/operator-sdk/internal/pkg/scorecard/scorecard.go:396 +0x8a
    github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).execute(0xc00061ea00, 0xc0009befd0, 0xb, 0xb, 0xc00061ea00, 0xc0009befd0)
            src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:762 +0x465
    github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0000f2780, 0x2053b60, 0xc0009e4e00, 0x0)
            src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:852 +0x2ec
    github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).Execute(...)
            src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:800
    main.main()
            src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:80 +0x4ce

Environment

• operator-sdk version:

v0.8.2

• go version:

go1.11.6

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+785687c", GitCommit:"785687c", GitTreeState:"clean", BuildDate:"2019-07-24T19:44:04Z", GoVersion:"go1.11.6", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

OCP v4.1.8 cluster

Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.4+4b86cc5", GitCommit:"4b86cc5", GitTreeState:"clean", BuildDate:"2019-07-24T19:42:25Z", GoVersion:"go1.11.6", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Scorecard command was run as part of the CVP operator pipeline and replicated locally using a testing OCP 4.1.8 cluster provisioned with AWS.

[estroz]

@dirgim the scorecard (incorrectly) assumes that if metadata.annotations["alm-examples"] is present it will be non-empty. Your CSV's corresponding field is empty, causing an index panic. I'll make a PR to fix this.

[dirgim]

@estroz great, thanks for looking into it!

[estroz]

@dirgim #1789 fixes the panic. Note that you will still have to add CR examples to metadata.annotations['alm-examples'] in your CSV or set the cr-manifest scorecard config option to run the scorecard like so:

$ cat .osdk-scorecard.yaml
cr-manifest:
  - deploy/crds/cache_v1alpha1_memcached_cr.yaml
  - deploy/crds/cache_v1alpha1_memcachedrs_cr.yaml