Move dev guide topics to separate guide (#4030)

docs/devguide/contributing.asciidoc

@@ -0,0 +1,141 @@
+[[beats-contributing]]
+== Contributing to Beats
+
+IMPORTANT: Please post all questions and issues first on
+https://discuss.elastic.co/c/beats[https://discuss.elastic.co/c/beats]
+before opening a Github Issue.
+
+The Beats are open source and we love to receive contributions from our
+community — you!
+
+There are many ways to contribute, from writing tutorials or blog posts,
+improving the documentation, submitting bug reports and feature requests, or
+writing code that implements a whole new protocol, module, or Beat.
+
+If you have a bugfix or new feature that you would like to contribute, please
+start by opening a topic on the https://discuss.elastic.co/c/beats[forums].
+It may be that somebody is already working on it, or that there are particular
+issues that you should know about before implementing the change.
+
+We enjoy working with contributors to get their code accepted. There are many
+approaches to fixing a problem and it is important to find the best approach
+before writing too much code.
+
+The process for contributing to any of the Elastic repositories is similar.
+
+[float]
+[[contribution-steps]]
+=== Contribution Steps
+
+. Please make sure you have signed our https://www.elastic.co/contributor-agreement/[Contributor License Agreement]. We are not asking you to assign
+copyright to us, but to give us the right to distribute your code without
+restriction. We ask this of all contributors in order to assure our users of the
+origin and continuing existence of the code. You only need to sign the CLA once.
+
+. Send a pull request! Push your changes to your fork of the repository and https://help.github.com/articles/using-pull-requests[submit a pull request]. In
+the pull request, describe what your changes do and mention any bugs/issues
+related to the pull request.
+
+[float]
+[[adding-new-beat]]
+=== Adding a New Beat
+
+If you want to create a new Beat, please read <<new-beat>>. You don't need to
+submit the code to this repository. Most new Beats start in their own repository
+and just make use of the libbeat packages. After you have a working Beat that
+you'd like to share with others, open a PR to add it to our list of 
+https://github.com/elastic/beats/blob/master/libbeat/docs/communitybeats.asciidoc[community
+Beats].
+
+[float]
+[[setting-up-dev-environment]]
+=== Setting Up Your Dev Environment
+
+The Beats are Go programs, so install the latest version of
+http://golang.org/[golang] if you don't have it already. The current Go version
+used for development is Golang {go-version}.
+
+The location where you clone is important. Please clone under the source
+directory of your `GOPATH`. If you don't have `GOPATH` already set, you can
+simply set it to your home directory (`export GOPATH=$HOME`).
+
+[source,shell]
+--------------------------------------------------------------------------------
+mkdir -p ${GOPATH}/src/github.com/elastic
+cd ${GOPATH}/src/github.com/elastic
+git clone https://github.com/elastic/beats.git
+--------------------------------------------------------------------------------
+
+NOTE: If you have multiple go paths, use `${GOPATH%%:*}` instead of `${GOPATH}`.
+
+Then you can compile a particular Beat by using the Makefile. For example, for
+Packetbeat:
+
+[source,shell]
+--------------------------------------------------------------------------------
+cd beats/packetbeat
+make
+--------------------------------------------------------------------------------
+
+Some of the Beats might have extra development requirements, in which case you'll find a
+CONTRIBUTING.md file in the Beat directory.
+
+[float]
+[[update-scripts]]
+=== Update scripts
+
+The Beats use a variety of scripts based on Python to generate configuration files
+and documentation. The command used for this is:
+
+[source,shell]
+--------------------------------------------------------------------------------
+make update
+--------------------------------------------------------------------------------
+
+This command has the following dependencies:
+
+* Python {python}
+* https://virtualenv.pypa.io/en/latest/[virtualenv] for Python
+
+Virtualenv can be installed with the command `easy_install virtualenv` or `pip install virtualenv`.
+More details can be found https://virtualenv.pypa.io/en/latest/installation.html[here].
+
+[float]
+[[running-testsuite]]
+=== Testing
+
+You can run the whole testsuite with the following command:
+
+[source,shell]
+--------------------------------------------------------------------------------
+make testsuite
+--------------------------------------------------------------------------------
+
+Running the testsuite has the following requirements:
+
+* Python {python}
+* Docker {docker}
+* Docker-compose {docker-compose}
+
+
+[float]
+[[documentation]]
+=== Documentation
+
+The documentation for each Beat is located under `{beatname}/docs` and is based
+on asciidoc. After changing the docs, you should verify that the docs are still
+building to avoid breaking the automated docs build. To build the docs run
+`make docs`. If you want to preview the docs for a specific Beat, run
+`make docs-preview` inside the folder for the Beat. This will automatically open
+your browser with the docs for preview.
+
+[float]
+[[dependencies]]
+=== Dependencies
+
+To manage the `vendor/` folder we use
+https://github.com/kardianos/govendor[govendor]. Please see
+the govendor documentation on how to add or update vendored dependencies.
+
+In most cases `govendor fetch your/dependency@version +out` will get the job done.
+

metricbeat/docs/developer-guide/create-metricset.asciidoc → docs/devguide/create-metricset.asciidoc

@@ -1,5 +1,5 @@
 [[creating-metricsets]]
-== Creating a Metricset
+=== Creating a Metricset
 
 A metricset is the part of a Metricbeat module that fetches and structures the
 data from the remote service. Each module can have multiple metricsets. In this guide, you learn how to create your own metricset. If you want to create
@@ -56,22 +56,24 @@ contains the following files:
 Let's look at the files in more detail next.
 
 [float]
-=== \{metricset}.go File
+==== \{metricset}.go File
 
 The first file is `{metricset}.go`. It contains the logic on how to fetch data from the service and convert it for sending to the output.
 
 The generated file looks like this:
 
+https://github.com/elastic/beats/blob/master/metricbeat/scripts/module/metricset/metricset.go.tmpl
+
 [source,go]
 ----
-include::../../scripts/module/metricset/metricset.go.tmpl[]
+include::../../metricbeat/scripts/module/metricset/metricset.go.tmpl[]
 ----
 
 The `package` clause and `import` declaration are part of the base structure of each Golang file. You should only
 modify this part of the file if your implementation requires more imports. 
 
 [float]
-==== Initialisation
+===== Initialisation
 
 The init method registers the metricset with the central registry. In Golang the `init()` function is called
 before the execution of all other code. This means the module will be automatically registered with the global registry.
@@ -88,7 +90,7 @@ func init() {
 ----
 
 [float]
-==== Definition
+===== Definition
 
 The MetricSet type defines all fields of the metricset. As a minimum it must inherit the `mb.BaseMetricSet` fields,
 but can be extended with additional entries. These variables can be used to persist data or configuration between
@@ -107,7 +109,7 @@ type MetricSet struct {
 
 
 [float]
-==== Creation
+===== Creation
 
 The `New` function creates a new instance of the MetricSet. The setup process
 of the MetricSet is also part of `New`. This method will be called before `Fetch`
@@ -135,7 +137,7 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
 ----
 
 [float]
-==== Fetching
+===== Fetching
 
 The `Fetch` method is the central part of the metricset. `Fetch` is called every
 time new data is retrieved. If more than one host is defined, `Fetch` is
@@ -170,7 +172,7 @@ https://godoc.org/github.com/elastic/beats/libbeat/common#MapStr[MapStr API docs
 
 
 [float]
-==== Multi Fetching
+===== Multi Fetching
 Metricbeat has two different `Fetch` interfaces. One of the interfaces, which you saw in the
 previous example, fetches a single event. In some cases, every fetch returns a list of events.
 Instead of using an array inside a JSON document, we recommend that you create a list of events.
@@ -186,10 +188,10 @@ The only difference between this and the previous example is that the second exa
 Metricbeat will add the same timestamp to all the events in the list to make it possible to correlate the events.
 
 [float]
-==== Parsing and Normalizing Fields
+===== Parsing and Normalizing Fields
 
 In Metricbeat we aim to normalize the metric names from all metricsets to
-respect a common {libbeat}/event-conventions.html[set of conventions]. This
+respect a common <<event-conventions,set of conventions>>. This
 makes it easy for users to find and interpret metrics. To simplify parsing,
 converting, renaming, and restructuring of the object read from the monitored
 system to the Metricbeat format, we have created the
@@ -254,7 +256,7 @@ In the above example, note that it is possible to create the schema object once
 and apply it to all events.
 
 [float]
-=== Configuration File
+==== Configuration File
 The configuration file for a metricset is handled by the module. If there are 
 multiple metricsets in one module, make sure you add all metricsets to the configuration.
 For example:
@@ -270,11 +272,13 @@ metricbeat:
 NOTE: Make sure that you run `make collect` after updating the config file
 so that your changes are also applied to the global configuration file and the docs.
 
-For more details about the configuration file, see <<configuration-metricbeat>>.
+For more details about the Metricbeat configuration file, see the topic about
+{metricbeat}/configuration-metricbeat.html[Modules] in the Metricbeat
+documentation.
 
 
 [float]
-=== What to Do Next
+==== What to Do Next
 This topic provides basic steps for creating a metricset. For more details about metricsets
 and how to extend your metricset further, see <<metricset-details>>.
 

metricbeat/docs/developer-guide/create-module.asciidoc → docs/devguide/create-module.asciidoc

@@ -1,5 +1,5 @@
 [[creating-metricbeat-module]]
-== Creating a Metricbeat Module
+=== Creating a Metricbeat Module
 
 Metricbeat modules are used to group multiple metricsets together and to implement shared functionality
 of the metricsets. In most cases, no implementation of the module is needed and the default module
@@ -9,7 +9,7 @@ It's important to complete the configuration and documentation files for a modul
 metricset by running `make create-metricset`, default versions of these files are generated in the `_meta` directory.
 
 [float]
-=== Module Files
+==== Module Files
 
 * `config.yml` and `config.full.yml`
 * `docs.asciidoc`
@@ -20,13 +20,13 @@ files are updated.
 
 
 [float]
-==== config.yml and config.full.yml
+===== config.yml and config.full.yml
 
 The `config.yml` file contains the basic configuration options and looks like this:
 
 [source,yaml]
 ----
-include::../../scripts/module/config.yml[]
+include::../../metricbeat/scripts/module/config.yml[]
 ----
 
 It contains the module name, your metricset, and the default period. If you have multiple
@@ -42,19 +42,19 @@ to add and document more advanced configuration options that should not be part
 config file shipped by default.
 
 [float]
-==== docs.asciidoc
+===== docs.asciidoc
 
 The `dosc.asciidoc` file contains the documentation about your module. During generation of the
 documentation, the default config file will be appended to the docs. Use this file to describe your
 module in more detail and to document specific configuration options.
 
 [source,asciidoc]
 ----
-include::../../scripts/module/docs.asciidoc[]
+include::../../metricbeat/scripts/module/docs.asciidoc[]
 ----
 
 [float]
-==== fields.yml
+===== fields.yml
 
 The `fields.yml` file contains the top level structure for the fields in your metricset. It's used in combination with
 the `fields.yml` file in each metricset to generate the template and documentation for the fields.
@@ -63,14 +63,14 @@ The default file looks like this:
 
 [source,yaml]
 ----
-include::../../scripts/module/fields.yml[]
+include::../../metricbeat/scripts/module/fields.yml[]
 ----
 
 Make sure that you update at least the description of the module.
 
 
 [float]
-=== Testing
+==== Testing
 
 It's a common pattern to use a `testing.go` file in the module package to share some testing functionality among
 the metricsets. This file does not have `_test.go` in the name because otherwise it would not be compiled for sub packages.

metricbeat/docs/developer-guide/creating-beat-from-metricbeat.asciidoc → docs/devguide/creating-beat-from-metricbeat.asciidoc

@@ -1,11 +1,11 @@
 [[creating-beat-from-metricbeat]]
-== Creating a Beat based on Metricbeat
+=== Creating a Beat based on Metricbeat
 
 The metricset Beat generator enables you to create a Beat that uses Metricbeat as a library and has your
 own metricsets.
 
 [float]
-=== Requirements
+==== Requirements
 
 To create your own Beat, you must have Golang {go-version} or later installed, and the `$GOPATH`
 must be set up correctly. In addition, the following tools are required:
@@ -16,7 +16,7 @@ must be set up correctly. In addition, the following tools are required:
 Virtualenv is easiest installed with your package manager or https://pip.pypa.io/en/stable/[pip].
 
 [float]
-=== Step 1 - Get the metricbeat source code
+==== Step 1 - Get the metricbeat source code
 
 The first step is to get the metricbeat source code:
 
@@ -41,7 +41,7 @@ This directory is normally located under `$GOPATH/src/github.com/{your-github-na
 
 
 [float]
-=== Step 2 - Create the Beat
+==== Step 2 - Create the Beat
 
 Run the command:
 
@@ -54,7 +54,7 @@ When prompted, enter the Beat name and path.
 
 
 [float]
-=== Step 3 - Init and create the metricset
+==== Step 3 - Init and create the metricset
 
 After creating the Beat, change the directory to `$GOPATH/src/github.com/{your-github-name}/{beat}` and run:
 
@@ -70,7 +70,7 @@ For more details about creating a metricset, see the docs about https://www.elas
 
 
 [float]
-=== Step 4 - Build & Run
+==== Step 4 - Build & Run
 
 To create a binary run the `make` command. This will create the binary in your beats directory.
 
@@ -84,7 +84,7 @@ To run it, execute the binary. This will automatically load the default configur
 This will run the beat with debug output enabled to the console to directly see what is happening. Stop the beat with `CTRL-C`.
 
 [float]
-=== Step 5 - Package
+==== Step 5 - Package
 
 To create packages and binaries for different plaforms, https://www.docker.com/[docker] is required.
 The first step is to get the most recent packaging tools into your beat: