go mod support for new operators

.travis.yml

@@ -7,7 +7,7 @@ cache:
     - $HOME/.cache/go-build
 
 go:
-- 1.10.3
+- 1.11.x
 
 # The `x_base_steps` top-level key is unknown to travis,
 # so we can use it to create a bunch of common build step

CHANGELOG.md

@@ -4,6 +4,7 @@
 
 - New option for [`operator-sdk build --image-builder`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#build), which can be used to specify which image builder to use. Adds support for [buildah](https://github.com/containers/buildah/). ([#1311](https://github.com/operator-framework/operator-sdk/pull/1311))
 - Manager is now configured with a new `DynamicRESTMapper`, which accounts for the fact that the default `RESTMapper`, which only checks resource types at startup, can't handle the case of first creating a CRD and then an instance of that CRD. ([#1329](https://github.com/operator-framework/operator-sdk/pull/1329))
+- [Go module](https://github.com/golang/go/wiki/Modules) support by default for new Go operators and during Ansible and Helm operator migration. The dependency manager used for a new operator can be explicitly specified for new operators through the `--dep-manager` flag, available in [`operator-sdk new`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#new) and [`operator-sdk migrate`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#migrate). `dep` is still available through `--dep-manager=dep`. ([#1001](https://github.com/operator-framework/operator-sdk/pull/1001))
 
 ### Changed
 
@@ -13,6 +14,7 @@
 - Renamed `--docker-build-args` option to `--image-build-args` option for `build` subcommand, because this option can now be shared with other image build tools than docker when `--image-builder` option is specified. ([#1311](https://github.com/operator-framework/operator-sdk/pull/1311))
 - Reduces Helm release information in CR status to only the release name and manifest and moves it from `status.conditions` to a new top-level `deployedRelease` field. ([#1309](https://github.com/operator-framework/operator-sdk/pull/1309))
   - **WARNING**: Users with active CRs and releases who are upgrading their helm-based operator should upgrade to one based on v0.7.0 before upgrading further. Helm operators based on v0.8.0+ will not seamlessly transition release state to the persistent backend, and will instead uninstall and reinstall all managed releases.
+- Go operator CRDs are overwritten when being regenerated by [`operator-sdk generate openapi`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#openapi). Users can now rely on `+kubebuilder` annotations in their API code, which provide access to most OpenAPIv3 [validation properties](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.0.md#schema-object) (the full set will be supported in the near future, see [this PR](https://github.com/kubernetes-sigs/controller-tools/pull/190)) and [other CRD fields](https://book.kubebuilder.io/beyond_basics/generating_crd.html). ([#1278](https://github.com/operator-framework/operator-sdk/pull/1278))
 
 ### Deprecated
 
@@ -23,12 +25,14 @@
 ### Bug Fixes
 
 - In Helm-based operators, when a custom resource with a failing release is reverted back to a working state, the `ReleaseFailed` condition is now correctly removed. ([#1321](https://github.com/operator-framework/operator-sdk/pull/1321))
+- [`operator-sdk generate openapi`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#openapi) no longer overwrites CRD values derived from `+kubebuilder` annotations in Go API code. See issues ([#1212](https://github.com/operator-framework/operator-sdk/issues/1212)) and ([#1323](https://github.com/operator-framework/operator-sdk/issues/1323)) for discussion. ([#1278](https://github.com/operator-framework/operator-sdk/pull/1278))
 
 ## v0.7.0
 
 ### Added
 
 - New optional flag `--header-file` for commands [`operator-sdk generate k8s`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#k8s) and [`operator-sdk add api`](https://github.com/operator-framework/operator-sdk/blob/master/doc/sdk-cli-reference.md#api) to supply a boilerplate header file for generated code. ([#1239](https://github.com/operator-framework/operator-sdk/pull/1239))
+- JSON output support for `operator-sdk scorecard` subcommand ([#1228](https://github.com/operator-framework/operator-sdk/pull/1228))
 
 ### Changed
 

Gopkg.toml

@@ -30,6 +30,12 @@
   name = "sigs.k8s.io/controller-runtime"
   version = "=v0.1.10"
 
+# This override revision has a fix that allows CRD unit tests to run correctly.
+# Remove once v0.1.11 is released.
+[[override]]
+  name = "sigs.k8s.io/controller-tools"
+  revision = "9d55346c2bde73fb3326ac22eac2e5210a730207"
+
 [[constraint]]
   name = "github.com/sergi/go-diff"
   version = "1.0.0"

cmd/operator-sdk/add/api.go

@@ -107,6 +107,7 @@ func apiRun(cmd *cobra.Command, args []string) error {
 
 	s := &scaffold.Scaffold{}
 	err = s.Execute(cfg,
+		&scaffold.Stub{Resource: r},
 		&scaffold.Types{Resource: r},
 		&scaffold.AddToScheme{Resource: r},
 		&scaffold.Register{Resource: r},

cmd/operator-sdk/build/cmd.go

@@ -184,6 +184,7 @@ func buildFunc(cmd *cobra.Command, args []string) error {
 			PackagePath: filepath.Join(projutil.CheckAndGetProjectGoPkg(), scaffold.ManagerDir),
 			Args:        goTrimFlags,
 			Env:         goBuildEnv,
+			GoMod:       projutil.IsDepManagerGoMod(),
 		}
 		if err := projutil.GoBuild(opts); err != nil {
 			return fmt.Errorf("failed to build operator binary: (%v)", err)
@@ -218,6 +219,7 @@ func buildFunc(cmd *cobra.Command, args []string) error {
 					PackagePath: testLocationBuild + "/...",
 					Args:        append(goTrimFlags, "-c"),
 					Env:         goBuildEnv,
+					GoMod:       projutil.IsDepManagerGoMod(),
 				},
 			}
 			if err := projutil.GoTest(opts); err != nil {

cmd/operator-sdk/migrate/cmd.go

@@ -41,7 +41,7 @@ func NewCmd() *cobra.Command {
 		RunE:  migrateRun,
 	}
 
-	newCmd.Flags().StringVar(&depManager, "dep-manager", "dep", `Dependency manager the new project will use (choices: "dep")`)
+	newCmd.Flags().StringVar(&depManager, "dep-manager", "modules", `Dependency manager the new project will use (choices: "dep", "modules")`)
 
 	return newCmd
 }
@@ -159,8 +159,10 @@ func scaffoldHelmDepManager(s *scaffold.Scaffold, cfg *input.Config) error {
 	switch m := projutil.DepManagerType(depManager); m {
 	case projutil.DepManagerDep:
 		files = append(files, &helm.GopkgToml{})
+	case projutil.DepManagerGoMod:
+		files = append(files, &helm.GoMod{}, &scaffold.Tools{})
 	default:
-		return projutil.ErrInvalidDepManager
+		return projutil.ErrInvalidDepManager(depManager)
 	}
 	return s.Execute(cfg, files...)
 }
@@ -170,8 +172,10 @@ func scaffoldAnsibleDepManager(s *scaffold.Scaffold, cfg *input.Config) error {
 	switch m := projutil.DepManagerType(depManager); m {
 	case projutil.DepManagerDep:
 		files = append(files, &ansible.GopkgToml{})
+	case projutil.DepManagerGoMod:
+		files = append(files, &ansible.GoMod{}, &scaffold.Tools{})
 	default:
-		return projutil.ErrInvalidDepManager
+		return projutil.ErrInvalidDepManager(depManager)
 	}
 	return s.Execute(cfg, files...)
 }

cmd/operator-sdk/new/cmd.go

@@ -54,7 +54,7 @@ generates a skeletal app-operator application in $GOPATH/src/github.com/example.
 	newCmd.Flags().StringVar(&apiVersion, "api-version", "", "Kubernetes apiVersion and has a format of $GROUP_NAME/$VERSION (e.g app.example.com/v1alpha1) - used with \"ansible\" or \"helm\" types")
 	newCmd.Flags().StringVar(&kind, "kind", "", "Kubernetes CustomResourceDefintion kind. (e.g AppService) - used with \"ansible\" or \"helm\" types")
 	newCmd.Flags().StringVar(&operatorType, "type", "go", "Type of operator to initialize (choices: \"go\", \"ansible\" or \"helm\")")
-	newCmd.Flags().StringVar(&depManager, "dep-manager", "dep", `Dependency manager the new project will use (choices: "dep")`)
+	newCmd.Flags().StringVar(&depManager, "dep-manager", "modules", `Dependency manager the new project will use (choices: "dep", "modules")`)
 	newCmd.Flags().BoolVar(&skipGit, "skip-git-init", false, "Do not init the directory as a git repository")
 	newCmd.Flags().BoolVar(&generatePlaybook, "generate-playbook", false, "Generate a playbook skeleton. (Only used for --type ansible)")
 	newCmd.Flags().BoolVar(&isClusterScoped, "cluster-scoped", false, "Generate cluster-scoped resources instead of namespace-scoped")
@@ -156,8 +156,10 @@ func doGoScaffold() error {
 	switch m := projutil.DepManagerType(depManager); m {
 	case projutil.DepManagerDep:
 		err = s.Execute(cfg, &scaffold.GopkgToml{})
+	case projutil.DepManagerGoMod:
+		err = s.Execute(cfg, &scaffold.GoMod{}, &scaffold.Tools{})
 	default:
-		err = projutil.ErrInvalidDepManager
+		err = projutil.ErrNoDepManager
 	}
 	if err != nil {
 		return fmt.Errorf("dependency manager file scaffold failed: (%v)", err)
@@ -382,8 +384,16 @@ func getDeps() error {
 		if err := execProjCmd("dep", "ensure", "-v"); err != nil {
 			return err
 		}
+	case projutil.DepManagerGoMod:
+		log.Info("Running go mod ...")
+		if err := execProjCmd("go", "mod", "vendor", "-v"); err != nil {
+			return err
+		}
+		if err := execProjCmd("go", "mod", "tidy", "-v"); err != nil {
+			return err
+		}
 	default:
-		return projutil.ErrInvalidDepManager
+		return projutil.ErrInvalidDepManager(depManager)
 	}
 	log.Info("Done getting dependencies")
 	return nil

cmd/operator-sdk/printdeps/cmd.go

@@ -33,15 +33,16 @@ func NewCmd() *cobra.Command {
 		Short: "Print Golang packages and versions required to run the operator",
 		Long: `The operator-sdk print-deps command prints all Golang packages and versions expected
 by this version of the Operator SDK. Versions for these packages should match
-those in an operators' Gopkg.toml file.
+those in an operators' go.mod or Gopkg.toml file, depending on the dependency
+manager chosen when initializing or migrating a project.
 
 print-deps prints in columnar format by default. Use the --as-file flag to
-print in Gopkg.toml file format.
+print in go.mod or Gopkg.toml file format.
 `,
 		RunE: printDepsFunc,
 	}
 
-	printDepsCmd.Flags().BoolVar(&asFile, "as-file", false, "Print dependencies in Gopkg.toml file format.")
+	printDepsCmd.Flags().BoolVar(&asFile, "as-file", false, "Print dependencies in go.mod or Gopkg.toml file format, depending on the dependency manager chosen when initializing or migrating a project")
 
 	return printDepsCmd
 }
@@ -72,14 +73,17 @@ func printDeps(asFile bool) error {
 		if isDep {
 			return ansible.PrintDepGopkgTOML(asFile)
 		}
+		return ansible.PrintGoMod(asFile)
 	case projutil.IsOperatorHelm():
 		if isDep {
 			return helm.PrintDepGopkgTOML(asFile)
 		}
+		return helm.PrintGoMod(asFile)
 	case projutil.IsOperatorGo():
 		if isDep {
 			return scaffold.PrintDepGopkgTOML(asFile)
 		}
+		return scaffold.PrintGoMod(asFile)
 	}
 
 	return projutil.ErrUnknownOperatorType{}

cmd/operator-sdk/scorecard/cmd.go

@@ -51,6 +51,7 @@ func NewCmd() *cobra.Command {
 	scorecardCmd.Flags().String(scorecard.ProxyImageOpt, fmt.Sprintf("quay.io/operator-framework/scorecard-proxy:%s", strings.TrimSuffix(version.Version, "+git")), "Image name for scorecard proxy")
 	scorecardCmd.Flags().String(scorecard.ProxyPullPolicyOpt, "Always", "Pull policy for scorecard proxy image")
 	scorecardCmd.Flags().String(scorecard.CRDsDirOpt, scaffold.CRDsDir, "Directory containing CRDs (all CRD manifest filenames must have the suffix 'crd.yaml')")
+	scorecardCmd.Flags().StringP(scorecard.OutputFormatOpt, "o", "human-readable", "Output format for results. Valid values: human-readable, json")
 	scorecardCmd.Flags().Bool(scorecard.VerboseOpt, false, "Enable verbose logging")
 
 	if err := viper.BindPFlags(scorecardCmd.Flags()); err != nil {

cmd/operator-sdk/test/local.go

@@ -213,6 +213,7 @@ func testLocalGoFunc(cmd *cobra.Command, args []string) error {
 			PackagePath: args[0] + "/...",
 			Env:         append(os.Environ(), fmt.Sprintf("%v=%v", test.TestNamespaceEnv, tlConfig.namespace)),
 			Dir:         projutil.MustGetwd(),
+			GoMod:       projutil.IsDepManagerGoMod(),
 		},
 		TestBinaryArgs: testArgs,
 	}

cmd/operator-sdk/up/local.go

@@ -183,8 +183,9 @@ func buildLocal(outputBinName string) error {
 	}
 	opts := projutil.GoCmdOptions{
 		BinName:     outputBinName,
-		PackagePath: filepath.Join(scaffold.ManagerDir, scaffold.CmdFile),
+		PackagePath: filepath.Join(projutil.CheckAndGetProjectGoPkg(), scaffold.ManagerDir),
 		Args:        args,
+		GoMod:       projutil.IsDepManagerGoMod(),
 	}
 	return projutil.GoBuild(opts)
 }

doc/ansible/user-guide.md

@@ -12,7 +12,7 @@ powered by Ansible using tools and libraries provided by the Operator SDK.
 - [ansible-runner][ansible_runner_tool] version v1.1.0+
 - [ansible-runner-http][ansible_runner_http_plugin] version v1.0.0+
 - [dep][dep_tool] version v0.5.0+. (Optional if you aren't installing from source)
-- [go][go_tool] version v1.10+. (Optional if you aren't installing from source)
+- [go][go_tool] version v1.11+. (Optional if you aren't installing from source)
 - Access to a Kubernetes v.1.9.0+ cluster.
 
 **Note**: This guide uses [minikube][minikube_tool] version v0.25.0+ as the
@@ -288,7 +288,7 @@ $ export OPERATOR_NAMESPACE=$(kubectl config view --minify -o jsonpath='{.contex
 $ sed -i "s|REPLACE_NAMESPACE|$OPERATOR_NAMESPACE|g" deploy/role_binding.yaml
 ```
 
-**Note**  
+**Note**
 If you are performing these steps on OSX, use the following commands instead:
 ```
 $ sed -i "" 's|{{ REPLACE_IMAGE }}|quay.io/example/memcached-operator:v0.0.1|g' deploy/operator.yaml
@@ -395,6 +395,18 @@ kubectl logs deployment/memcached-operator -c operator
 The `ansible` logs contain all of the information about the Ansible run and will make it much easier to debug issues within your Ansible tasks,
 whereas the `operator` logs will contain much more detailed information about the Ansible Operator's internals and interface with Kubernetes.
 
+### Additional Ansible debug
+
+Occasionally while developing additional debug in the Operator logs is nice to have. To enable Ansible debug output, ie `-vvvv`.
+Add the following to the `operator.yaml` manifest.
+
+```yaml
+          env:
+           ...
+           - name: ANSIBLE_VERBOSITY
+             value: "4"
+```
+
 ### Update the size
 
 Change the `spec.size` field in the memcached CR from 3 to 4 and apply the

doc/dev/developer_guide.md

@@ -5,7 +5,7 @@ This document explains how to setup your dev environment.
 ## Prerequisites
 - [dep][dep_tool] version v0.5.0+
 - [git][git_tool]
-- [go][go_tool] version v1.10+
+- [go][go_tool] version v1.11+
 
 ## Download Operator SDK
 

doc/dev/release.md

@@ -171,7 +171,7 @@ Create a new branch to push release commits:
 $ git checkout -b release-v1.3.0
 ```
 
-Commit changes to the following six files:
+Commit changes to the following files:
 
 - `version/version.go`: update `Version` to `v1.3.0`.
 - `internal/pkg/scaffold/gopkgtoml.go`, under the `[[constraint]]` for `github.com/operator-framework/operator-sdk`:
@@ -181,6 +181,12 @@ Commit changes to the following six files:
 - `internal/pkg/scaffold/gopkgtoml_test.go`: same as for `internal/pkg/scaffold/gopkgtoml.go`.
 - `internal/pkg/scaffold/ansible/gopkgtoml.go`: same as for `internal/pkg/scaffold/gopkgtoml.go`.
 - `internal/pkg/scaffold/helm/gopkgtoml.go`: same as for `internal/pkg/scaffold/gopkgtoml.go`.
+- `internal/pkg/scaffold/go_mod.go`, in the `replace` block for `github.com/operator-framework/operator-sdk`:
+  - Add the following `replace` entry: `github.com/operator-framework/operator-sdk => github.com/operator-framework/operator-sdk v1.3.0`.
+  - If an entry already exists, change the version to `v1.3.0`.
+- `internal/pkg/scaffold/go_mod_test.go`: same as for `internal/pkg/scaffold/go_mod.go`.
+- `internal/pkg/scaffold/helm/go_mod.go`: same as for `internal/pkg/scaffold/go_mod.go`.
+- `internal/pkg/scaffold/ansible/go_mod.go`: same as for `internal/pkg/scaffold/go_mod.go`.
 - `CHANGELOG.md`: update the `## Unreleased` header to `## v1.3.0`.
 
 Create a new PR for `release-v1.3.0`.

doc/project_layout.md

@@ -10,9 +10,10 @@ The `operator-sdk` CLI generates a number of packages for each project. The foll
 | pkg/controller | This pkg contains the controller implementations. Users are expected to edit the `pkg/controller/<kind>/<kind>_controller.go` to define the controller's reconcile logic for handling a resource type of the specified `kind`. |
 | build | Contains the `Dockerfile` and build scripts used to build the operator. |
 | deploy | Contains various YAML manifests for registering CRDs, setting up [RBAC][RBAC], and deploying the operator as a Deployment.
-| Gopkg.toml Gopkg.lock | The [Go Dep][dep] manifests that describe the external dependencies of this operator. |
+| (Gopkg.toml Gopkg.lock) or (go.mod go.sum) | The [Go mod][go_mod] or [Go Dep][dep] manifests that describe the external dependencies of this operator, depending on the dependency manager chosen when initializing or migrating a project. |
 | vendor | The golang [vendor][Vendor] folder that contains the local copies of the external dependencies that satisfy the imports of this project. [Go Dep][dep] manages the vendor directly. |
 
 [RBAC]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
 [Vendor]: https://golang.org/cmd/go/#hdr-Vendor_Directories
+[go_mod]: https://github.com/golang/go/wiki/Modules
 [dep]: https://github.com/golang/dep