Make geoip steps generic and move to libbeat (#10947)

* Make geoip steps generic and move to libbeat

* Add changes from review

* Add topic to all the beats

auditbeat/docs/configuring-howto.asciidoc

@@ -28,6 +28,7 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-dashboards>>
@@ -62,6 +63,8 @@ include::./auditbeat-filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]

filebeat/docs/configuring-howto.asciidoc

@@ -33,6 +33,7 @@ The following topics describe how to configure Filebeat:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-dashboards>>
@@ -72,6 +73,8 @@ include::./filebeat-filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]

heartbeat/docs/configuring-howto.asciidoc

@@ -29,6 +29,7 @@ The following topics describe how to configure Heartbeat:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-dashboards>>
@@ -59,6 +60,8 @@ include::./heartbeat-filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]

journalbeat/docs/configuring-howto.asciidoc

@@ -20,6 +20,7 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-template>>
@@ -48,6 +49,8 @@ include::./filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]

libbeat/docs/shared-geoip.asciidoc

@@ -0,0 +1,142 @@
+[id="{beatname_lc}-geoip"]
+== Enrich events with geoIP information
+
+ifeval::["{beatname_lc}"=="packetbeat"]
+TIP: To populate the client locations map in the {beatname_uc} dashboard, follow
+the steps in this section.
+endif::[]
+
+You can use {beatname_uc} along with the {ref}/geoip-processor.html[GeoIP
+Processor] in {es} to export geographic location information based on IP
+addresses. Then you can use this information to visualize the location of IP
+addresses on a map in {kib}.
+
+The `geoip` processor adds information about the geographical location of
+IP addresses, based on data from the Maxmind GeoLite2 City Database. Because the
+processor uses a geoIP database that's installed on {es}, you don't need
+to install a geoIP database on the machines running {beatname_uc}.
+
+NOTE: If your use case involves using {ls}, you can use the
+{logstash-ref}/plugins-filters-geoip.html[GeoIP filter] available in {ls}
+instead of using the `geoip` processor. However, using the `geoip` processor is
+the simplest approach when you don't require the additional processing power of
+{ls}.
+
+[float]
+[id="{beatname_lc}-configuring-geoip"]
+=== Configure the `geoip` processor
+
+To configure {beatname_uc} and the `geoip` processor:
+
+1. Define an ingest node pipeline that uses one or more `geoip` processors to
+add location information to the event. For example, you can use the Console in
+{kib} to create the following pipeline:
++
+--
+[source,json]
+----
+PUT _ingest/pipeline/geoip-info
+{
+  "description": "Add geoip info",
+  "processors": [
+    {
+      "geoip": {
+        "field": "client.ip",
+        "target_field": "client.geo",
+        "ignore_missing": true
+      }
+    },
+    {
+      "geoip": {
+        "field": "source.ip",
+        "target_field": "source.geo",
+        "ignore_missing": true
+      }
+    },
+    {
+      "geoip": {
+        "field": "destination.ip",
+        "target_field": "destination.geo",
+        "ignore_missing": true
+      }
+    },
+    {
+      "geoip": {
+        "field": "server.ip",
+        "target_field": "server.geo",
+        "ignore_missing": true
+      }
+    },
+    {
+      "geoip": {
+        "field": "host.ip",
+        "target_field": "host.geo",
+        "ignore_missing": true
+      }
+    }
+  ]
+}
+----
+//CONSOLE
+--
++
+In this example, the pipeline ID is `geoip-info`. `field` specifies the field
+that contains the IP address to use for the geographical lookup, and
+`target_field` is the field that will hold the geographical information.
+`"ignore_missing": true` configures the pipeline to continue processing when
+it encounters an event that doesn't have the specified field.
++
+See
+{ref}/geoip-processor.html[GeoIP Processor] for more options.
++
+To learn more about adding host information to an event, see
+<<add-host-metadata>>.
+
+2. In the {beatname_uc} config file, configure the {es} output to use the
+pipeline. Specify the pipeline ID in the `pipeline` option under
+`output.elasticsearch`. For example:
++
+[source,yaml]
+-------------------------------------------------------------------------------
+output.elasticsearch:
+  hosts: ["localhost:9200"]
+  pipeline: geoip-info
+-------------------------------------------------------------------------------
+
+3. Run {beatname_uc}. Remember to use `sudo` if the config file is owned by
+root.
++
+["source","sh",subs="attributes"]
+-------------------------------------------------------------------------------
+./{beatname_lc} -e
+-------------------------------------------------------------------------------
++
+If the lookups succeed, the events are enriched with `geo_point` fields, such as
+`client.geo.location` and `host.geo.location`, that you can use to populate
+visualizations in {kib}.
+
+ifeval::["{beatname_lc}"=="packetbeat"]
+As a convenience, the {beatname_uc} index template already has mappings defined
+for `client.geo.location`, `source.geo.location`, `destination.geo.location`,
+`server.geo.location`, and `host.geo.location`. The mappings ensure that each
+field, when it exists, gets indexed as a `geo_point`.
+endif::[]
+
+If you add a field that's not already defined as a `geo_point` in the
+index template, add a mapping so the field gets indexed correctly. 
+
+[float]
+[id="{beatname_lc}-visualizing-location"]
+=== Visualize locations
+
+To visualize the location of IP addresses, you can
+ifdef::has_map[]
+<<load-kibana-dashboards,set up the example {kib} dashboards>> (if
+you haven't already), or
+endif::has_map[]
+create a new {kibana-ref}/tilemap.html[coordinate map] in {kib} and select the
+location field, for example `client.geo.location` or `host.geo.location`, as
+the Geohash.
+
+[role="screenshot"]
+image:./images/coordinate-map.png[Coordinate map in {kib}]

metricbeat/docs/configuring-howto.asciidoc

@@ -30,6 +30,7 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-dashboards>>
@@ -62,6 +63,8 @@ include::./metricbeat-filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]

packetbeat/docs/configuring-howto.asciidoc

@@ -30,7 +30,7 @@ The following topics describe how to configure Packetbeat:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
-* <<packetbeat-geoip>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-dashboards>>
@@ -59,7 +59,7 @@ include::./packetbeat-filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
-include::./packetbeat-geoip.asciidoc[]
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 

packetbeat/docs/index.asciidoc

@@ -14,6 +14,7 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[]
 :discuss_forum: beats/{beatname_lc}
 :beat_default_index_prefix: {beatname_lc}
 :has_ml_jobs: yes
+:has_map:
 :deb_os:
 :rpm_os:
 :mac_os:

winlogbeat/docs/configuring-howto.asciidoc

@@ -25,6 +25,7 @@ The following topics describe how to configure Winlogbeat:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<configuration-path>>
 * <<setup-kibana-endpoint>>
 * <<configuration-dashboards>>
@@ -53,6 +54,8 @@ include::./winlogbeat-filtering.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 include::{libbeat-dir}/docs/shared-path-config.asciidoc[]
 
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]

x-pack/functionbeat/docs/configuring-howto.asciidoc

@@ -20,6 +20,7 @@ The following topics describe how to configure {beatname_uc}:
 * <<configuration-ssl>>
 * <<filtering-and-enhancing-data>>
 * <<configuring-ingest-node>>
+* <<{beatname_lc}-geoip>>
 * <<setup-kibana-endpoint>>
 * <<configuration-template>>
 * <<configuration-logging>>
@@ -53,6 +54,9 @@ include::./filtering.asciidoc[]
 [role="xpack"]
 include::{libbeat-dir}/docs/shared-config-ingest.asciidoc[]
 
+[role="xpack"]
+include::{libbeat-dir}/docs/shared-geoip.asciidoc[]
+
 [role="xpack"]
 include::{libbeat-dir}/docs/shared-kibana-config.asciidoc[]