Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] Basic Elastic Agent documentation #19030

Merged
merged 50 commits into from
Jun 10, 2020
Merged
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
846b029
docs
michalpristas Jun 7, 2020
4135449
relative links
michalpristas Jun 8, 2020
66bd030
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
9d7aee1
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
f6a4a7d
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
f2673e0
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
0003890
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
f0a9dc1
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
de8baa4
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
250e8e7
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
ce87631
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
b2d2657
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
84c2ea9
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
d321304
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
d6197fa
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
90f0980
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
780c4aa
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
c5522a4
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
8e6ea52
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
96a3ff5
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
4cc6312
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
1950232
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
1bc2c22
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
371d8ed
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
d946757
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
c7e162b
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
62bc73f
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
1fa71f1
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
5fd8923
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
87266ad
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
75b43c7
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
d210096
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
4e2221d
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
335ebe8
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
67d1877
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
6e6a4ab
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
4abc7f8
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
cd875fd
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
f17b3f3
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
bf55bf6
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
97c416b
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
c6a499e
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
7539961
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
71a5e81
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 9, 2020
69c6e23
wip
michalpristas Jun 9, 2020
8a437af
default behavior notice
michalpristas Jun 9, 2020
37cab7b
Update x-pack/elastic-agent/docs/elastic-agent.asciidoc
michalpristas Jun 10, 2020
683fe89
step1
michalpristas Jun 10, 2020
816e5b0
Merge branch 'agent-first-docs' of github.com:michalpristas/beats int…
michalpristas Jun 10, 2020
d53710b
mod
michalpristas Jun 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
320 changes: 242 additions & 78 deletions x-pack/elastic-agent/docs/elastic-agent.asciidoc
Original file line number Diff line number Diff line change
@@ -1,78 +1,242 @@
Agent's notes


[[requirements]]
Requirements

The remote Kibana version of the Agent must be equal or higher than the Agent version, if this is not met

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@michalpristas I think the requirement is that the Agent must be equal or higher than the Agent minor version. I think we are still able to make patch releases for Agent without updating the whole stack. Can you confirm?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we removed sending kbn-version as there was a check on kibana side which required it to be == when sent. not sure what the status of the check is atm cc @nchaulet

Kibana will refuse the connection.


[[Supported Actions by the Agent]]

The Elastic Agent supports the following actions received by Fleet.

Implemented:

[[Configuration change]]

The Elastic Agent receives a configuration encoded as a JSON object.

[source,json]
------------------------------------------------------------------------------
{
"type": "CONFIG_CHANGE",
"id": "id1",
"data": {
"config": {
"id": "policy-id",
"outputs": {
"default": {
"hosts": "https://localhost:9200"
}
},
"datasources": [{
"id": "string",
"enabled": true,
"use_output": "default",
"inputs": [{
"type": "logs",
"streams": [{
"paths": ["/var/log/hello.log"]
}]
}]
}]
}
}
}
------------------------------------------------------------------------------

[[Planned but not implemented]]

[source,json]
------------------------------------------------------------------------------
{
"type": "UPGRADE",
"id": "id1",
"target_version": "7.2.1"
}
------------------------------------------------------------------------------

[source,json]
------------------------------------------------------------------------------
{
"type": "DOWNGRADE",
"id": "id1",
"target_version": "7.2.1"
}
------------------------------------------------------------------------------

[source,json]
------------------------------------------------------------------------------
{
"type": "RELAY",
"id": "id1",
"destination": {"name":"endpoint"}
"data": {}
}
------------------------------------------------------------------------------
[[elastic-agent-installation-configuration]]
== Get started with {beatname_uc}

++++
<titleabbrev>Get started</titleabbrev>
++++

Elastic Agent is a single, unified agent that you can deploy to hosts or containers to collect data and send it to the {stack}. Behind the scenes, Elastic Agent runs the {beats} shippers or Endpoint required for your configuration.

* <<elastic-agent-installation>>
* <<elastic-agent-execution-modes>>
* <<elastic-agent-cmd-options>>
* <<elastic-agent-configuration>>

[[elastic-agent-installation]]
== Install Elastic Agent

=== Step 1: Unpack archive


[[mac]]
*mac:*
dedemorton marked this conversation as resolved.
Show resolved Hide resolved

ifeval::["{release-state}"=="unreleased"]

Version {version} of {beatname_uc} has not yet been released.

endif::[]

ifeval::["{release-state}"!="unreleased"]

["source","sh",subs="attributes,callouts"]
------------------------------------------------
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{version}-darwin-x86_64.tar.gz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit odd to have step 1 and the curl command. I'd choose one or the other, not both.

tar xzvf elastic-agent-{version}-darwin-x86_64.tar.gz
------------------------------------------------

endif::[]

[[linux]]
*linux:*

ifeval::["{release-state}"=="unreleased"]

Version {version} of {beatname_uc} has not yet been released.

endif::[]

ifeval::["{release-state}"!="unreleased"]

["source","sh",subs="attributes,callouts"]
------------------------------------------------
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-{version}-linux-x86_64.tar.gz
tar xzvf elastic-agent-{version}-linux-x86_64.tar.gz
------------------------------------------------

endif::[]

[[win]]
*win:*

ifeval::["{release-state}"=="unreleased"]

Version {version} of {beatname_uc} has not yet been released.

endif::[]

ifeval::["{release-state}"!="unreleased"]

. Download the Elastic Agent Windows zip file from the
https://www.elastic.co/downloads/beats/elastic-agent[downloads page].

. Extract the contents of the zip file into `C:\Program Files`.

. Rename the `elastic-agent-<version>-windows` directory to `Elastic-Agent`.

. Open a PowerShell prompt as an Administrator (right-click the PowerShell icon and select *Run As Administrator*).

. From the PowerShell prompt, run the following commands to install Filebeat as a
Windows service:
+
[source,shell]
----------------------------------------------------------------------
PS > cd 'C:\Program Files\Elastic-Agent'
PS C:\Program Files\Elastic-Agent> .\install-service-elastic-agent.ps1
----------------------------------------------------------------------

NOTE: If script execution is disabled on your system, you need to set the execution policy for the current session to allow the script to run. For example: `PowerShell.exe -ExecutionPolicy UnRestricted -File .\install-service-elastic-agent.ps1`.

endif::[]

=== Step 2: Run Elastic Agent

If Elastic Agent is not installed as an auto-starting service, start it manually:


[source,shell]
----------------------------------------------------------------------
$ ./elastic-agent run
----------------------------------------------------------------------

[[elastic-agent-execution-modes]]
== Execution modes
dedemorton marked this conversation as resolved.
Show resolved Hide resolved

Elastic Agent runs in two modes: standalone or fleet. The two modes differ in how you configure and manage the agent.
[float]
=== Standalone mode

With _standalone mode_, you manually configure and manage the agent locally. Each agent is configured to be in standalone mode by default after installation.
At startup, Elastic Agent reads the configuration file specified by the `-c` argument or uses the default configuration, `elastic-agent.yml`, which is located in the same directory as the agent.

For configuration options see `elastic-agent_configuration_example.yml`

=== Fleet mode

With _fleet mode_, you manage Elastic Agent remotely. The agent uses a trusted {kib} instance to retrieve configurations and report agent events. This trusted {kib} instance must have Ingest Manager and Fleet enabled.

To create a trusted communication channel between Elastic Agent and {kib}, you enroll the agent to Fleet.

To enroll an Elastic Agent to Fleet:


michalpristas marked this conversation as resolved.
Show resolved Hide resolved
. Stop the agent.

michalpristas marked this conversation as resolved.
Show resolved Hide resolved
. Enroll the agent:
+
[source,shell]
----------------------------------------------------------------------
$ ./elastic-agent http://localhost:5601 $token
dedemorton marked this conversation as resolved.
Show resolved Hide resolved
----------------------------------------------------------------------
+
`$token` is an enrollment token acquired from Fleet.

[[elastic-agent-cmd-options]]
== Command line options

The `elastic-agent run` command provides flags that alter the behavior of an agent.

==== `-path.home`
dedemorton marked this conversation as resolved.
Show resolved Hide resolved

The home directory of the Elastic Agent. `path.home` determines the location of the configuration files and data directory.

==== `-c`

The configuration file to load.
If not specified, Elastic Agent uses `{path.home}/elastic-agent.yml`.


==== `-path.data`

The data directory used by Elastic Agent to store downloaded artifacts. Also stores logs for any Beats started and managed by Elastic Agent.

If not specified, Elastic Agent uses `{path.home}/data`.

[[elastic-agent-configuration]]
== Configure Elastic Agent

By default Elastic Agent runs in standalone mode to ingest system data and send it to a local {es} instance running on port 9200. It uses the demo credentials of the `elastic` user. It's also configured to monitor all Beats managed by the agent and send the Beats logs and metrics to the same {es) instance.

To alter this behavior, configure the output.

=== Configure the output

michalpristas marked this conversation as resolved.
Show resolved Hide resolved
Elastic Agent enables definition of multiple outputs where each data source can be paired with different output.
Copy link
Contributor

@dedemorton dedemorton Jun 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can say this more concisely. How about:

Define one or more outputs. You can pair each data source with a different output. For example:

There are still a couple of problems with this. Will users know what a data source is? What do you mean by pairing?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, my comment had some typos. Fixed. :-P


At the moment Elastic Agent works only with Elasticsearch output.
Sample configuration can look like the example below:

[source,yaml]
-------------------------------------------------------------------------------------
outputs:
default:
type: elasticsearch
hosts: [127.0.0.1:9200]
username: elastic
password: changeme
monitoring:
type: elasticsearch
api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw
hosts: ["localhost:9200"]
ca_sha256: "7lHLiyp4J8m9kw38SJ7SURJP4bXRZv/BNxyyXkCcE/M="
-------------------------------------------------------------------------------------

This example configures two outputs: `default` and `monitoring`.
Notice that they use different authentication methods. The first one uses a username and password pair, and the second one contains an api key.

[NOTE]
==============
A default output configuration is required.
==============

=== Configure Beats monitoring

michalpristas marked this conversation as resolved.
Show resolved Hide resolved
Elastic Agent is monitoring _Beats_ by default. To disable or change monitoring settings, set options under `settings.monitoring`:

[source,yaml]
-------------------------------------------------------------------------------------
settings.monitoring:
# enabled turns on monitoring of running processes
enabled: true
# enables log monitoring
logs: true
# enables metrics monitoring
metrics: true
# specifies output to be used
use_output: monitoring
-------------------------------------------------------------------------------------


To disable monitoring, set `settings.monitoring.enabled` to `false`. When set to `false`, Beats monitoring is turned off, and all other options in this section are ignored.
To enable monitoring, set `settings.monitoring.enabled` to `true`. Also set the `logs` and `metrics` settings to control whether logs, metrics, or both are collected. If neither setting is specified, monitoring is disabled.


Set `use_output` to specify the output to which monitoring events are sent.

=== Specify data sources

By default Elastic Agent collects system metrics, such as cpu, memory, network, and filesystem metrics, and sends them to the default output. For example:


[source,yaml]
-------------------------------------------------------------------------------------
datasources:
- namespace: default
use_output: default
inputs:
- type: system/metrics
streams:
- metricset: cpu
dataset: system.cpu
- metricset: memory
dataset: system.memory
- metricset: network
dataset: system.network
- metricset: filesystem
dataset: system.filesystem
-------------------------------------------------------------------------------------

If `use_output` is not specified, the `default` output is used.

For more examples, see `elastic-agent_configuration_example.yml`