Skip to content

Commit

Permalink
[docs] Update ingest management docs with 7.9 UI changes (elastic#80)
Browse files Browse the repository at this point in the history
* [docs] Update ingest management docs with 7.9 UI changes

* Update ingest management docs with other 7.9 changes

* Reference local copy of js/css

* Fix bug in platform selector

* Minor edit to match agent docs
  • Loading branch information
dedemorton committed Aug 8, 2020
1 parent 0098d2c commit f5926cc
Show file tree
Hide file tree
Showing 25 changed files with 382 additions and 181 deletions.
230 changes: 93 additions & 137 deletions docs/en/ingest-management/getting-started.asciidoc
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
[[ingest-management-getting-started]]
[role="xpack"]
= Get started with ingest management
= Quick start: Get logs and metrics into the {stack}

:release-state: released

++++
<titleabbrev>Get started</titleabbrev>
<titleabbrev>Quick start</titleabbrev>
++++

beta[]

This guide describes how to get started with the new {ingest-management}
capabilities available in this release.
This guide describes how to use the {ingest-management} capabilities in the
{stack} to:

* Collect logs and metrics from systems and services across your organization
* Send the data to the {stack}
* Explore and visualize the data in real-time

For feedback and questions, please contact us in the {im-forum}[discuss forum].

Expand All @@ -19,121 +25,70 @@ For feedback and questions, please contact us in the {im-forum}[discuss forum].

Before you begin, please read <<ingest-management-limitations>>.

To use this **beta** release of {ingest-management}, you need:

* An {es} cluster and {kib} (version 7.8) with a basic license.
You can use our https://www.elastic.co/cloud/elasticsearch-service[hosted {ess}]
on {ecloud}, or {stack-gs}/get-started-elastic-stack.html[install the {stack} on
your own hardware].

* A user with the superuser role. See {ref}/built-in-roles.html[Built-in roles].

* On self-managed clusters, you must configure security and encryption settings.
If you're using our https://www.elastic.co/cloud/elasticsearch-service[hosted {ess}]
on {ecloud}, these settings are already enabled.
You need {es} for storing and searching your data, and {kib} for visualizing and
managing it. You can use our
https://www.elastic.co/cloud/elasticsearch-service[hosted {ess}]
on {ecloud} (recommended), or self-manage the {stack} on your own hardware.

** In your {es} configuration:
+
- {ref}/configuring-security.html[Configure {es} security]. Set
`xpack.security.enabled` to `true`.
- {ref}/security-settings.html#api-key-service-settings[Enable the built-in API key service].
Set `xpack.security.authc.api_key.enabled` to `true`.
Here's what you need for each deployment type:

** In your {kib} configuration:
+
- {kibana-ref}/using-kibana-with-security.html[Configure {kib} security]. Set
`xpack.security.enabled` to `true`.
- {kibana-ref}/configuring-tls.html[Configure TLS]. As an alternative, you can
disable the TLS check by setting `xpack.ingestManager.fleet.tlsCheckDisabled`
to `true`. For example, you might want to disable TLS checking if {kib} is
behind a proxy that terminates the SSL connection.
- Set `xpack.encryptedSavedObjects.encryptionKey` to any alphanumeric value of
at least 32 characters. For example:
`xpack.encryptedSavedObjects.encryptionKey: "something_at_least_32_characters"`. {fleet}
requires this setting in order to save API keys and encrypt them in {kib}.

[discrete]
[[enable-ingest-management]]
== Step 1: Enable {ingest-management}

To enable {ingest-management}:

. Add the following setting to the {kib} configuration:
+
[source,yaml]
----
xpack.ingestManager.enabled: true
----
+
* On our
https://www.elastic.co/cloud/elasticsearch-service[hosted {ess}] on
{ecloud}, add the setting to the {kib} user settings as described in
{cloud}/ec-manage-kibana-settings.html[Add {kib} user settings].
* On a self-managed stack, add the setting to the {kib} configuration as
described in
{kibana-ref}/ingest-manager-settings-kb.html[{ingest-manager} settings].

. Verify that {ingest-management} has been correctly enabled:
.. Log in to {kib} and go to **Management > {ingest-manager}**.
+
If the menu entry is visible, and you see the following start page
for {ingest-manager}, {ingest-management} is successfully enabled.
+
[role="screenshot"]
image::images/kibana-ingest-manager-start.png[{ingest-manager} app in {kib}]

//TODO: Add tabbed panel when the code is stable.
include::tab-widgets/prereq-widget.asciidoc[]

[discrete]
[[install-integration]]
== Step 2: Install an integration and create a data source
== Step 1: Add integrations

{ingest-manager} provides _integrations_ that bundle various assets needed to
ingest and visualize data.

In this guide, we assume that you have `nginx` running on some of your
infrastructure, and want to collect logs and metrics from it. To do so:

. In the {ingest-manager} app, click the **Integrations** tab and use the
search bar to find the Nginx integration.
. Log in to {kib} and go to **Management > {ingest-manager}**.
+
[role="screenshot"]
image::images/kibana-ingest-manager-integrations-nginx.png[{ingest-manager} app showing Nginx Integration]
image::images/kibana-ingest-manager-start.png[{ingest-manager} in {kib}]

. In {ingest-manager}, click the **Integrations** tab and use the search bar to
find the Nginx integration.
+
[role="screenshot"]
image::images/kibana-ingest-manager-integrations-nginx.png[{ingest-manager} showing Nginx integration]
. Click the Nginx integration to see more details about it, and then click
**Create data source**.
**Add Nginx**.
+
[role="screenshot"]
image::images/kibana-ingest-manager-integrations-nginx-overview.png[{ingest-manager} app showing Nginx Integration overview]
image::images/kibana-ingest-manager-integrations-nginx-overview.png[{ingest-manager} showing Nginx integration overview]

. On the **Add data source** page, click the default Agent configuration
. On the **Add Nginx integration** page, click the default agent configuration
to select it, then scroll down to inspect or change the default settings.

. If your logs are not in the default location, click the down arrow next to
enabled streams and change the **Paths** field.

. When you're done, click **Save data source**.
. When you're done, click **Save integration**.
+
[role="screenshot"]
image::images/kibana-ingest-manager-integrations-nginx-add-datasource.png[{ingest-manager} app Add data source screen]
image::images/kibana-ingest-manager-integrations-nginx-add-datasource.png[{ingest-manager} Add data source screen]

. To verify that the integration is installed, click the **Integrations** tab
and then click **Installed Integrations**.
+
The Nginx integration has been installed and should show up in this list. Note
that the **System** and **Elastic Endpoint** integrations are installed by
default.
that the **System** and **Elastic Endpoint Security** integrations are installed
by default.
+
[role="screenshot"]
image::images/kibana-ingest-manager-integrations-list-installed.png[{ingest-manager} app showing installed integrations]
image::images/kibana-ingest-manager-integrations-list-installed.png[{ingest-manager} showing installed integrations]

. Select the **Configurations** tab, and in the Agent configurations list, click
the default config.
. Select the **Configurations** tab, and in the list of agent configurations,
click **Default config**.
+
The newly created data source should appear on the **Data sources** tab.
Note that the `system-1` data source has been created by default.
The newly added Nginx integration should appear under **Ingegrations**.
Note that the `system-1` integration has been created by default.
+
[role="screenshot"]
image::images/kibana-ingest-manager-configurations-default-with-nginx.png[{ingest-manager} app showing default Agent configuration with nginx-1 datasource]
image::images/kibana-ingest-manager-configurations-default-with-nginx.png[{ingest-manager} showing default agent configuration with nginx-1 datasource]

[discrete]
[[install-run-elastic-agent]]
Expand All @@ -143,95 +98,93 @@ include::{beats-repo-dir}/x-pack/elastic-agent/docs/elastic-agent.asciidoc[tag=a

include::{beats-repo-dir}/x-pack/elastic-agent/docs/install-elastic-agent.asciidoc[tag=install-elastic-agent]

To configure {agent}, you can use the {ingest-manager} app in {kib} (see
<<agent-fleet-mode,{fleet} mode>>), or configure it manually (see
<<agent-standalone-mode, Standalone mode>>).
See the https://www.elastic.co/downloads/elastic-agent[download page] for other
installation options.

After the download is complete, decide which mode you want to run in: {fleet}
or standalone. What's the difference?

* With <<agent-fleet-mode,_{fleet} mode_>>, you use {ingest-manager} in {kib} to
define and manage {agent} configurations from a central location.
* With <<agent-standalone-mode,_standalone mode_>>, you configure {agent}
manually on the system where the agent is installed.

[discrete]
[[agent-fleet-mode]]
=== {fleet} mode

To use {fleet} to configure {agent}:

. Enable {fleet} in the {ingest-manager} app. To do so, click the **{fleet}**
tab and click **Create user and enable Fleet**.
. Enable {fleet} in {ingest-manager}. To do so, click the **{fleet}**
tab and click **Create user and enable {fleet}**.
+
[role="screenshot"]
image::images/kibana-ingest-manager-fleet-enable.png[{ingest-manager} app showing prompt to enable {fleet}]
image::images/kibana-ingest-manager-fleet-enable.png[{ingest-manager} showing prompt to enable {fleet}]

. On the **{fleet}** tab, click **Add agent** and complete the steps under
**Enroll with {fleet}**. Skip the first step because you downloaded {agent}
earlier.

. Select an agent configuration (or accept the default).

. On the **Fleet** tab, click **Enroll new agent** to start the enrollment.
Select an Agent configuration (or accept the default) and copy the command for
enrolling the Agent.
. Copy the command for enrolling the agent.
+
The copied command contains your {kib} URL and an enrollment key that was
generated by the {ingest-manager} app.
generated by {ingest-manager}.
+
[role="screenshot"]
image::images/kibana-ingest-manager-fleet-enrol.png[{ingest-manager} app showing Agent enrollment dialog]
image::images/kibana-ingest-manager-fleet-enrol.png[{ingest-manager} showing agent enrollment dialog]

. In the directory where you installed {agent}, paste the command to enroll the
Agent. Note that this command will overwrite the `elastic-agent.yml` file in
that directory.
. In the directory where {agent} is installed, run the command you copied. Note
that this command will overwrite the `elastic-agent.yml` file in that directory.
+
["source","sh",subs="attributes"]
----
./elastic-agent enroll KIBANA_URL ENROLLMENT_KEY
This will replace your current settings. Do you want to continue? [Y/n]:
----
--
include::{beats-repo-dir}/x-pack/elastic-agent/docs/tab-widgets/enroll-widget.asciidoc[]
--

. Run the Agent:
. Run {agent}:
+
--
// tag::run-agent[]
[source,shell]
----
./elastic-agent run <1>
----
<1> On Windows, you must run {agent} under the SYSTEM account if you plan
to use the {elastic-endpoint} integration.
// end::run-agent[]
include::{beats-repo-dir}/x-pack/elastic-agent/docs/tab-widgets/run-widget.asciidoc[]
--

. In the {ingest-manager} app, click **Continue** to go to the **{fleet}**
tab, where you should see the newly enrolled Agent.
. In {ingest-manager}, click **Continue** to go to the **{fleet}**
tab, where you should see the newly enrolled agent.
+
[role="screenshot"]
image::images/kibana-ingest-manager-fleet-agents.png[{ingest-manager} app showing enrolled Agents]
image::images/kibana-ingest-manager-fleet-agents.png[{ingest-manager} showing enrolled agents]

If you run into problems, see <<ingest-management-troubleshooting>>.

TIP: To unenroll an Agent, choose **Unenroll** from the **Actions**
menu for the Agent. This will invalidate the API key the Agent uses to
connect to {es}. The {agent} will continue to run, but will not be able to send
data. It will show this error instead:
`invalid api key to authenticate with fleet`.
TIP: To unenroll an agent, see <<unenroll-elastic-agent>>.

[discrete]
[[agent-standalone-mode]]
=== Standalone mode (manual configuration)

To configure {agent} manually:

. In the {ingest-manager} app, click the **Configurations** tab, and in the
Agent configurations list, click the default config.
. On the **{fleet}** tab, click **Add agent** and complete the steps under
**Standalone mode**. Skip the first step because you downloaded {agent}
earlier.

. Select an agent configuration (or accept the default).

. Select the **YAML** tab to see the configuration for {agent}. Copy the
content and put it into a file named `elastic-agent.yml` on the
system where {agent} is installed.
. Copy the content of the agent configuration and put it into a file named
`elastic-agent.yml` on the system where {agent} is installed.
+
[role="screenshot"]
image::images/kibana-ingest-manager-configurations-default-yaml.png[{ingest-manager} app showing default Agent configuration in YAML format]
image::images/kibana-ingest-manager-configurations-default-yaml.png[{ingest-manager} showing default agent configuration in YAML format]
+
NOTE: The configuration file generated by the {ingest-manager} app already
NOTE: The configuration file generated by {ingest-manager} already
contains the correct {es} address and port for your setup. If you run everything
locally, the address is `127.0.0.1:9200`. If you use our
https://www.elastic.co/cloud/elasticsearch-service[hosted {ess}] on
{ecloud}, the address corresponds to the {es} endpoint URL that is listed
under **Endpoints** as described in
{cloud}/ec-working-with-elasticsearch.html[Work with {es}].

. Add your {es} username and password to the `outputs` section in the
. Change the {es} username and password in the `outputs` section of the
configuration file:
+
[source,yaml]
Expand All @@ -252,26 +205,29 @@ datasources:
. Run {agent}:
+
--
include::getting-started.asciidoc[tag=run-agent]
include::{beats-repo-dir}/x-pack/elastic-agent/docs/tab-widgets/run-standalone-widget.asciidoc[]
--

[discrete]
[[view-data]]
== Step 4: View your data

In the {ingest-manager} app, click the **Data streams** tab to inspect the data
that is sent by the Agent. From the **Actions** column, you can navigate to the
In {ingest-manager}, click the **Datasets** tab to inspect the data
that is sent by {agent}. From the **Actions** column, you can navigate to the
dashboards corresponding to the data type that is sent.

[role="screenshot"]
image::images/kibana-ingest-manager-datastreams.png[{ingest-manager} app showing data streams list]
image::images/kibana-ingest-manager-datastreams.png[{ingest-manager} showing data streams list]

//Adding this section for future use. Might be premature to add this for the
//beta release.

//[discrete]
//== What's next?
[discrete]
== What's next?

Now that you have your logs streaming into {es}, learn how to unify your logs,
metrics, uptime, and application performance data by using
https://www.elastic.co/observability[Elastic {observability}].

//Now that you have your logs streaming into {es}, learn how to unify your logs,
//metrics, uptime, and application performance data. Use the Observability apps in
//{kib} to search across all your data.
// Add Javascript and CSS for tabbed panels
include::tab-widgets/code.asciidoc[]
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/en/ingest-management/images/data-source.png
Binary file not shown.
Binary file removed docs/en/ingest-management/images/data-streams.png
Binary file not shown.
Binary file added docs/en/ingest-management/images/datasets.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/en/ingest-management/images/integrations.png
100755 → 100644
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/en/ingest-management/images/kibana-ingest-manager-start.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/en/ingest-management/index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
:doctype: book
:beats-repo-dir: {beats-root}
:im-forum: https://discuss.elastic.co/tags/c/elastic-stack/81/stack-ingest-management
:code-path: {beats-repo-dir}/libbeat/docs/tab-widgets/code

= Ingest Management Guide

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,13 @@ to use {fleet}, the {agent}s must have a direct network connection to {kib} and
{es}. It is also possible to run the {agent}s in standalone mode in cases where
a network connection is not available or not needed.

This feature has several additional limitations at the current time:
This feature has additional limitations at the current time:

* Support for only 9 integrations (more coming soon)
* Support for only {filebeat} and {metricbeat}
* We recommend you enroll no more than 100 Agents
* Recommended limit of no more than 100 enrolled agents
* No output to {ls}, Kafka, or other remote clusters
* No proxy support in {agent}
* {kib} must have internet access to download integration packages
* Requires internet access for {kib} to download integration packages
* No support for advanced {beats} settings like multiline, processors, and so
on
* No custom index templates or pipelines
Expand Down
Loading

0 comments on commit f5926cc

Please sign in to comment.