Harvest Plugins

[cgrinds]

Harvest Plugins

Harvest v21.05.01 has limited support for dynamically linked Go plugins built using buildmode=plugin. Unfortunately Go's support for dynamic plugins is weak and comes with significant drawbacks.

Basically, Go plugins were not designed as a way for other people to extend your app. They were designed for you to extend your app.

Before outlining the pros and cons of Go's plugins, let's explore why we want them.

Why do we want Harvest plugins?

1. Plugins allow 3rd parties to extend Harvest without (re)building it. You program to Harvest's API and, at runtime, Harvest dynamically loads your code into its process and calls it.

2. You only "pay" for what you use. If you don't use a feature from plugin A, you don't pay for it in disk footprint, you don't load the code into memory, etc. This is less important than Update changelog and notice for 21.05.0 #1.

Plugins, as a concept, are great. They allow us to build a loosely-coupled modular system. Harvest's current implementation doesn't address 1 or 2, and arguably introduces more problems than it solves.

Cons of Go Plugins

1. Plugins and Harvest must be complied with the exact same version of Go.
2. Plugins and Harvest must be compiled with the same GOPATH.
3. Any imported packages between Harvest and the plugin must be the exact same version.
4. Plugins and Harvest can NOT vendor dependencies. If either have vendored dependencies, Harvest won't work.
5. Debuggers don't work with dynamic code - this means you can't use a debugger with Harvest right now because the interesting parts of Harvest are implemented as Go plugins.
6. Makes cross compiling harder or impossible (see Windows, Alpine, etc.)
7. Creates ~3x larger executables - with buildmode=plugin the bin directory is 140 MB, without buildmode=plugin the bin directory is 48M
8. ~7x slower builds

# with buildmode=plugin
$ make clean
$ time GOOS=darwin make build
Executed in   24.94 secs

# without buildmode=plugin
$ make clean
$ time GOOS=darwin make build
Executed in    3.39 secs

Experience of others

We're not the only team to hit issues with Go plugins. I haven't found a project that recommends them.

Traefik

Traefik tried and abandoned plugins due to development pain

traefik/traefik#1336 (comment)
traefik/traefik#1336 (comment)

OK, bad news...
We can't load an external plugin if Traefik is built with CGO_ENABLED=0, and we really need this to build a statically linked golang executable to run in a Docker container.
golang/go#19569 and there are no plan on this golang/go#19569 (comment)

If you compile traefik binary on your laptop, and a plugin in docker on your laptop, it does not work either: Error loading plugin: error opening plugin: plugin.Open: plugin was built with a different version of package

Ultimately they had so many problems they abandoned Go plugins and built a Go interpreter and use that instead.

Telegraf

Telegraf added Go plugin support, but they consider it experimental with limited support and it still requires a custom build of Telegraf. Their issue tracker has the usual build and version issues everyone does.

influxdata/telegraf#7162

Prometheus and VictoriaMetrics

Not sure if they rejected or never tried. Probably rejected.

Caddy

Not sure if Caddy learned from others and rejected Go plugins or went a different way from the beginning. The way you extend Caddy is by building a custom version yourself with side-effect loading init functions. No dynamic linking. Edit Caddy main and include imports.

Options for Harvest

1. Remove buildmode=plugin code. If folks want to extend Harvest with plugins they clone, add their code, and build their own version of Harvest.

2. Add an approach similar to Caddy and Benthos - build your plugin and add an import to Harvest's main . Benthos example

3. Keep what we have - not great given the downsides: no debugger, bigger executables, no cross compile.

4. Add some sort of exec model where we can call any process and read/write to stdout/stdin. Not great from a performance or security point of view. A poor man's RPC.

5. RPC - something like Hashicorp's. Performance may be a concern here too - the trick with both of these last two is to avoid too many trips across the RPC layer.

Recommendation

We should go with #1 & #2. Keep the plugin concept in Harvest, but don't implement it with Go plugins. First we remove buildmode=plugin (already done) and then we work on #2 using an architecture similar to Caddy's.

Until we implement #2, if you want to extend Harvest, you extend it the same way you would most open-source projects: clone, make your changes, keep your fork up to date.

Resources

• Is anyone actually using go plugins? Jun 2019
• Go plugins Nov 2020
• Things to avoid while using Golang plugins Nov 2018
• On Golang, "static" binaries, cross-compiling and plugins May 2017

[cgrinds]

Part 1 done, Part 2 WIP

[rahulguptajss]

can this be closed?

[cgrinds]

there's still some work to be completed here. It's not pressing though so I'll remove P1

[cgrinds]

We opted for option #1. Closing