Module arch improvement

[vgratian]

First, to safe us the language-confusion, about the terms used below:

• plugin - the plugins that Harvest runs as part of a collector
• module - a directory with several files that self-identify as a Go "package", this can be an exporter, collector, plugin or something else.

This is a follow-up on the changes applied by #106 for issue #100. This changes have greatly simplified Harvest and removed some unnecessary complications due to our use of Go's plugin library (not to be confused with Harvest's plugins). On the other hand, there are a number of drawbacks:

• List of collectors is hardcoded in the import section of cmd/poller/poller.go: if a user or developer wants to add a new collector, they have to edit that file (and re-edit after each update of Harvest).
• List of exporters is hardcoded in several sections of cmd/poller/poller.go
• List of plugins is hardcoded in several modules
• The gist of the new module-design is written in cmd/plugin/plugin.go. This module originally defined the plugin type and interfaces and implemented its base type ("AbstractPlugin"). Combining two unrelated concepts, will lead to unnecessary confusion in the future.
• Only collectors are implemented to use the plugin.RegisterModule() method

How I propose to improve:

• All Harvest modules that implement a collector, exporter or plugin are a defined in modules.txt. Users can safely edit this file to add new modules, including third-party modules.
• Running make modules will generate the file cmd/poller/modules/modules.go which will import all of these modules. This command will also optionally install these modules if they are not part of Harvest (e.g. third-party collectors).
• A separate module handles registering modules: cmd/poller/registrar/registrar.go. Each module will explicitly register either as a collector, plugin or exporter using the methods registrar.RegisterCollector(), registrar.RegisterExporter(), registrar.RegisterPlugin().
• Since the arguments of these functions are fixed, no type-checking is necessary at initialization-time.
• Modules don't need to declare additional types, like HarvestModule, for the same reason.
• Poller will initialize these modules similar to what is done currently with collectors before but with more simplicity.

I think these improvements will give Harvest a lot of flexibility. Adding new collectors, exporters or plugins does not require to touch the core code-base of Harvest. We only need to edit harvest.yml and modules.txt. I have even tried using this "third-party" collector this way.

Only issue left to address:

• If collector is from a third-party repository, you still have to manually copy the template to 'conf/...'. But I think this can be easily solved as well. (I didn't touch this issue, because I think we have to revisit how templates are loaded anyway).

[cgrinds]

@vgratian Can you incorporate the changes we discussed in our last meeting?

• investigate using go generate instead of echoes
• investigate if there's a good way to have modules consume extra config from harvest.yml
• consolidate concepts, pick one of: extension, plugin, module, etc.

I think your PR is the next natural step for what was started in #100.

[cgrinds]

Left general comment above

[cgrinds]

Closing this for now until we have time to pickup