Go modules #15853

kvch · 2020-01-27T09:43:55Z

This PR introduces the new dependency management system for beats. From now on we are using go mod instead of govendor. The name of the module we provide is named github.com/elastic/beats/v7.

We adopted go modules, but we still keep dependencies under the folder vendor. However, it is maintained by go mod now by running mage vendor. Thus, it is not possible to add local changes to the dependencies there because next time someone runs the command it will be overwritten. If you need to apply patches to dependencies either fork the repository or try to contribute it back to the original repo.

This PR does not address the changes in Beat generators. Those are going to be updated in a follow-up PR.

FAQ

How can I add a new dependency?

TL;DR

go get {{ module_name }}@{{ required_version }}
mage vendor
make notice

Run go get {{ module_name }}@{{ required_version }} in the repository. This adds the module to the requirements list in go.mod. In order to add the dependency to the vendor folder, run mage vendor. If the dependency contains files which are not copied by this command (e.g. C source files), add them to this list: https://github.com/elastic/beats/tree/master/dev-tools/mage/gomod.go

How can I upgrade a dependency?

The process is similar to adding a new dependency. Except for one step. If the dependency is updated to a newer major version, make sure to follow up changes in the root import paths of those libs. I suggest using the tool named mod to upgrade it: https://github.com/marwan-at-work/mod

How can I use a fork of a dependency?

If you need to use a fork of a dependency, add it to go.mod either manually or by running the following command:

go mod edit -replace {{ dependency_name }}={{ fork_name }}@{{ fork_version }}

Why is the major version part of the import path?

In a nutshell, it is needed to tell apart major version changes. Their import paths are different because those are incompatible. See more: https://blog.golang.org/v2-go-modules

Thus, when we are releasing v8.0.0, we need to change the import path again. The tool named mod is going to be useful for us in this case, too.

Why is `make check` failing?

Are you seeing the following error?

make[1]: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats'
diff --git a/go.mod b/go.mod
index 170f6660..fc5fb6ee 100644
--- a/go.mod
+++ b/go.mod
@@ -53,6 +53,7 @@ require (
 	github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6
 	github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
 	github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2
+	github.com/elastic/beats v7.6.0+incompatible
 	github.com/elastic/ecs v1.4.0
 	github.com/elastic/go-libaudit v0.4.0
 	github.com/elastic/go-licenser v0.2.1

If your go.mod lists github.com/elastic/beats as a requirement, the repo contains an outdated root import path. Change github.com/elastic/beats to github.com/elastic/beats/v7.

What is next?

After this PR is merged, please rebase all of your open PRs. In most cases rebasing will lead to conflicts. Possible sources of conflicts:

New import path conflicts with the old. To resolve it switch every github.com/elastic/beats import to github.com/elasitc/beats/v7.
A new dependency has been added or existing was updated. Update the dependency using go modules.

Related issues

#15868

kvch · 2020-02-24T14:48:57Z

jenkins test this

kvch · 2020-02-26T12:53:50Z

run beats-ci/package

kvch · 2020-02-26T12:58:02Z

run beats-ci/package

urso · 2020-02-28T15:32:21Z

One more tip: run go mod tidy before commiting go.mod. go get puts every dependency that have been seen into your go.mod. Some tools (e.g. vim-go) can add unwanted entries.

btw. do we have a make check target that checks if go.mod is tidy?

kvch · 2020-02-28T15:42:56Z

No, I haven't added a check, because mage vendor runs tidy for you. So if you have done anything to vendor, e.g. update or add a new one, go mod tidy is called. You think it is necessary to add it on top of this?

urso · 2020-02-28T16:28:33Z

Its not about packages we want to vendor. It is about people installing tools that are actually no dependency. When I install a custom tool (or vim-go does because I want to update some tools) while being in the Beats repository, then go will automatically add this tool as a dependency to the go.mod file, which will stay there if I don't call go mod vendor by chance. Not sure if this behavior changed in go1.14, but I managed to pollute my go.mod files a few times with this :)

kvch · 2020-03-02T08:38:35Z

run beats-ci/package

kvch · 2020-03-02T09:18:31Z

run beats-ci/package

kvch · 2020-03-02T12:30:03Z

Issue with node_integration_test.go of redis in x-pack/metricbeat is unrelated to this change.

CHANGELOG.next.asciidoc

This PR introduces the new dependency management system for beats. From now on we are using `go mod` instead of `govendor`. The name of the module we provide is named `github.com/elastic/beats/v7`. We adopted go modules, but we still keep dependencies under the folder `vendor`. However, it is maintained by `go mod` now by running `mage vendor`. Thus, it is not possible to add local changes to the dependencies there because next time someone runs the command it will be overwritten. **If you need to apply patches to dependencies either fork the repository or try to contribute it back to the original repo.** This PR does not address the changes in Beat generators. Those are going to be updated in a follow-up PR. TL;DR ```sh go get {{ module_name }}@{{ required_version }} mage vendor make notice ``` Run `go get {{ module_name }}@{{ required_version }}` in the repository. This adds the module to the requirements list in `go.mod`. In order to add the dependency to the vendor folder, run `mage vendor`. If the dependency contains files which are not copied by this command (e.g. C source files), add them to this list: https://github.com/elastic/beats/tree/master/dev-tools/mage/gomod.go The process is similar to adding a new dependency. Except for one step. If the dependency is updated to a newer major version, make sure to follow up changes in the root import paths of those libs. I suggest using the tool named `mod` to upgrade it: https://github.com/marwan-at-work/mod If you need to use a fork of a dependency, add it to `go.mod` either manually or by running the following command: ```sh go mod edit -replace {{ dependency_name }}={{ fork_name }}@{{ fork_version }} ``` In a nutshell, it is needed to tell apart major version changes. Their import paths are different because those are incompatible. See more: https://blog.golang.org/v2-go-modules Thus, when we are releasing v8.0.0, we need to change the import path again. The tool named mod is going to be useful for us in this case, too. Are you seeing the following error? ``` make[1]: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats' diff --git a/go.mod b/go.mod index 170f6660..fc5fb6ee 100644 --- a/go.mod +++ b/go.mod @@ -53,6 +53,7 @@ require ( github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4 github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2 + github.com/elastic/beats v7.6.0+incompatible github.com/elastic/ecs v1.4.0 github.com/elastic/go-libaudit v0.4.0 github.com/elastic/go-licenser v0.2.1 ``` If your `go.mod` lists `github.com/elastic/beats` as a requirement, the repo contains an outdated root import path. Change `github.com/elastic/beats` to `github.com/elastic/beats/v7`. After this PR is merged, please rebase all of your open PRs. In most cases rebasing will lead to conflicts. Possible sources of conflicts: 1. New import path conflicts with the old. To resolve it switch every `github.com/elastic/beats` import to `github.com/elasitc/beats/v7`. 2. A new dependency has been added or existing was updated. Update the dependency using go modules. - [Golang blog: Using Go modules](https://blog.golang.org/using-go-modules) - [Golang wiki](https://github.com/golang/go/wiki/Modules) elastic#15868 (cherry picked from commit 95626b8)

* Go modules (#15853) This PR introduces the new dependency management system for beats. From now on we are using `go mod` instead of `govendor`. The name of the module we provide is named `github.com/elastic/beats/v7`. We adopted go modules, but we still keep dependencies under the folder `vendor`. However, it is maintained by `go mod` now by running `mage vendor`. Thus, it is not possible to add local changes to the dependencies there because next time someone runs the command it will be overwritten. **If you need to apply patches to dependencies either fork the repository or try to contribute it back to the original repo.** This PR does not address the changes in Beat generators. Those are going to be updated in a follow-up PR. TL;DR ```sh go get {{ module_name }}@{{ required_version }} mage vendor make notice ``` Run `go get {{ module_name }}@{{ required_version }}` in the repository. This adds the module to the requirements list in `go.mod`. In order to add the dependency to the vendor folder, run `mage vendor`. If the dependency contains files which are not copied by this command (e.g. C source files), add them to this list: https://github.com/elastic/beats/tree/master/dev-tools/mage/gomod.go The process is similar to adding a new dependency. Except for one step. If the dependency is updated to a newer major version, make sure to follow up changes in the root import paths of those libs. I suggest using the tool named `mod` to upgrade it: https://github.com/marwan-at-work/mod If you need to use a fork of a dependency, add it to `go.mod` either manually or by running the following command: ```sh go mod edit -replace {{ dependency_name }}={{ fork_name }}@{{ fork_version }} ``` In a nutshell, it is needed to tell apart major version changes. Their import paths are different because those are incompatible. See more: https://blog.golang.org/v2-go-modules Thus, when we are releasing v8.0.0, we need to change the import path again. The tool named mod is going to be useful for us in this case, too. Are you seeing the following error? ``` make[1]: Leaving directory `/home/travis/gopath/src/github.com/elastic/beats' diff --git a/go.mod b/go.mod index 170f6660..fc5fb6ee 100644 --- a/go.mod +++ b/go.mod @@ -53,6 +53,7 @@ require ( github.com/dop251/goja_nodejs v0.0.0-20171011081505-adff31b136e6 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4 github.com/eclipse/paho.mqtt.golang v1.2.1-0.20200121105743-0d940dd29fd2 + github.com/elastic/beats v7.6.0+incompatible github.com/elastic/ecs v1.4.0 github.com/elastic/go-libaudit v0.4.0 github.com/elastic/go-licenser v0.2.1 ``` If your `go.mod` lists `github.com/elastic/beats` as a requirement, the repo contains an outdated root import path. Change `github.com/elastic/beats` to `github.com/elastic/beats/v7`. After this PR is merged, please rebase all of your open PRs. In most cases rebasing will lead to conflicts. Possible sources of conflicts: 1. New import path conflicts with the old. To resolve it switch every `github.com/elastic/beats` import to `github.com/elasitc/beats/v7`. 2. A new dependency has been added or existing was updated. Update the dependency using go modules. - [Golang blog: Using Go modules](https://blog.golang.org/using-go-modules) - [Golang wiki](https://github.com/golang/go/wiki/Modules) #15868 (cherry picked from commit 95626b8) * update generated file

bschaeffer · 2020-04-30T18:24:12Z

👋 New to go modules here.... why do all the tagged versions remove the go.mod? If I try to require github.com/elastic/beats v7.6.2 it just complains about a missing go.mod, which is present in master but none of the tagged releases.

jhump · 2020-07-22T02:22:43Z

@bschaeffer, FYI, there was no go.mod file at version v7.6.2. This PR wasn't merged until after that. Looks like the first release with it is v7.7.0. Also, with Go modules, a major version suffix is required when the major version is >1, e.g. /v7.

So you have to do something like this to get it to work:

require github.com/elastic/beats/v7 v7.7.0

kvch added enhancement [zube]: In Progress Team:Beats labels Jan 27, 2020

andresrc added [zube]: Inbox [zube]: In Progress and removed [zube]: In Progress [zube]: Inbox labels Jan 28, 2020

jsoriano mentioned this pull request Feb 17, 2020

Make generators to use govendor #3900

Closed

kvch marked this pull request as ready for review February 26, 2020 14:29

kvch requested review from a team as code owners February 26, 2020 14:29

kvch force-pushed the go-modules branch 3 times, most recently from 91077fd to b7b6bd6 Compare February 27, 2020 15:54

ChrsMark mentioned this pull request Feb 28, 2020

Prometheus remote write endpoint #16609

Merged

kvch force-pushed the go-modules branch from 3dbefdb to cc275ad Compare February 28, 2020 14:34

kvch force-pushed the go-modules branch from ede0689 to b687095 Compare March 2, 2020 08:38

kvch force-pushed the go-modules branch from 5be27a8 to 0227238 Compare March 2, 2020 09:14

urso reviewed Mar 2, 2020

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

kvch added 11 commits March 3, 2020 06:42

More fixes after rebase to upstream/master

4e50e09

Do not downgrade go-ucfg to 0.8.2

075ec15

Add minor fixes

3c16e1d

Use out fork of insomniacslk/dhcp to make Packetbeat work on Windows

7442948

Use packetbeat branch of github.com/elastic/dhcp

ad761c5

Disable checks for generators on Jenkins

6e573bd

Follow-up after rebasing on top of master i.

77a61ae

Make sure no dependency is added accidentally

1c98e7c

Import fixes after rebase

4e63927

Clean up changelog

a09f149

Fixes after rebase

8ab305e

kvch force-pushed the go-modules branch from 647417f to 8ab305e Compare March 3, 2020 06:08

kvch merged commit 95626b8 into master Mar 3, 2020

zube bot added [zube]: Done and removed [zube]: In Progress labels Mar 3, 2020

kvch mentioned this pull request Mar 3, 2020

Cherry-pick #15853 to 7.x: Go modules #16748

Merged

kvch added the v7.7.0 label Mar 3, 2020

kvch mentioned this pull request Mar 3, 2020

Cherry-pick #15853 to 7.6: Go modules #16749

Closed

kvch added the v7.6.1 label Mar 3, 2020

andresrc added Team:Integrations Label for the Integrations team and removed [zube]: Done labels Mar 6, 2020

kvch mentioned this pull request Mar 28, 2020

mage vendor chokes when go mod vendor deletes C source files. #17304

Closed

ghost mentioned this pull request Feb 8, 2024

Enhancement: Update go module version to v8 #37920

Closed

5 tasks

mengzhuo mentioned this pull request Feb 8, 2024

Enhancement: Update module name to v8 #37921

Closed

5 tasks

vivekschauhan mentioned this pull request Feb 21, 2024

Go module upgrade to v8 #38079

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Go modules #15853

Go modules #15853

kvch commented Jan 27, 2020 •

edited

Loading

kvch commented Feb 24, 2020

kvch commented Feb 26, 2020

kvch commented Feb 26, 2020

urso commented Feb 28, 2020

kvch commented Feb 28, 2020

urso commented Feb 28, 2020

kvch commented Mar 2, 2020

kvch commented Mar 2, 2020

kvch commented Mar 2, 2020 •

edited

Loading

bschaeffer commented Apr 30, 2020

jhump commented Jul 22, 2020

Go modules #15853

Go modules #15853

Conversation

kvch commented Jan 27, 2020 • edited Loading

FAQ

How can I add a new dependency?

How can I upgrade a dependency?

How can I use a fork of a dependency?

Why is the major version part of the import path?

Why is make check failing?

What is next?

Further reading for the interested

Related issues

kvch commented Feb 24, 2020

kvch commented Feb 26, 2020

kvch commented Feb 26, 2020

urso commented Feb 28, 2020

kvch commented Feb 28, 2020

urso commented Feb 28, 2020

kvch commented Mar 2, 2020

kvch commented Mar 2, 2020

kvch commented Mar 2, 2020 • edited Loading

bschaeffer commented Apr 30, 2020

jhump commented Jul 22, 2020

kvch commented Jan 27, 2020 •

edited

Loading

Why is `make check` failing?

kvch commented Mar 2, 2020 •

edited

Loading