Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Creates a data frame examples page #389

Merged
merged 33 commits into from
Jul 2, 2019
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1eccf7e
Added dataframes intro file to the ML section, amended overview.ascii…
szabosteve May 29, 2019
e4a91a3
Fixed paragraph style.
szabosteve May 29, 2019
ca68a7a
Fixed markup.
szabosteve May 29, 2019
016d36f
Fixed markup.
szabosteve May 29, 2019
1916513
Rephrased sentences to improve readability.
szabosteve May 29, 2019
65f6dab
Fixed header.
szabosteve May 30, 2019
a564230
Extended the text: pivoting, aggregation, continuous data frames.
szabosteve May 31, 2019
236ba3a
Amended the promise about continuous data frames.
szabosteve May 31, 2019
04051ba
Fixed typo.
szabosteve May 31, 2019
7e89296
Added simple example to the intro.
szabosteve May 31, 2019
00e6e92
Quick fix.
szabosteve May 31, 2019
a09d80f
Amended the text based on the technical and peer reviews.
szabosteve Jun 11, 2019
09246fe
Added beta tag to the page.
szabosteve Jun 11, 2019
648cb7d
Improve readability.
szabosteve Jun 12, 2019
0ed95f8
Adds screenshot to the example.
szabosteve Jun 12, 2019
17fb7cd
Merge branch 'master' of github.com:elastic/stack-docs
szabosteve Jun 13, 2019
59d2520
Merge branch 'master' of github.com:elastic/stack-docs
szabosteve Jun 13, 2019
5fb3569
Merge branch 'master' of github.com:elastic/stack-docs
szabosteve Jun 17, 2019
a80e070
Merge branch 'master' of github.com:elastic/stack-docs
szabosteve Jun 24, 2019
c93b440
Merge branch 'master' of github.com:elastic/stack-docs
szabosteve Jun 26, 2019
1bafd48
[DOCS] Creates a data frame examples page.
szabosteve Jun 26, 2019
a90c521
Fixes titleabbrev.
szabosteve Jun 26, 2019
f79da15
Adds sample response to the web log example.
szabosteve Jun 26, 2019
6a18931
[DOCS] Shortens data frame examples navigation title
lcawl Jun 27, 2019
6e16553
Fine-tunes the web log example.
szabosteve Jun 27, 2019
ea3c148
Adds best customer example to the page.
szabosteve Jun 27, 2019
3e0f07d
Adds Kibana sample data reference to the example intro.
szabosteve Jun 27, 2019
498ed35
Adds airline example section.
szabosteve Jun 27, 2019
bc03c8c
Adds the flight example to the example pool.
szabosteve Jun 27, 2019
3da0fec
Fixes typos.
szabosteve Jun 27, 2019
9e54139
Addresses feedback.
szabosteve Jun 28, 2019
d2e55c9
Addresses feedback.
szabosteve Jul 1, 2019
80ea756
Addresses feedback.
szabosteve Jul 2, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
161 changes: 161 additions & 0 deletions docs/en/stack/data-frames/dataframe-examples.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
[role="xpack"]
[testenv="basic"]
[[dataframe-examples]]
== {dataframe-cap} examples
++++
<titleabbrev>Examples</titleabbrev>
++++

beta[]

This page provides examples of how to use {dataframe-transforms} to derive useful
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
insights from your data. For a more detailed, step-by-step example, see
<<ecommerce-dataframes,Transforming your data with {dataframes}>>.

* <<example-clientips>>


[float]
[[example-clientips]]
=== Finding suspicious client IPs by using scripted metrics

With {dataframe-transforms}, you can use
{ref}/search-aggregations-metrics-scripted-metric-aggregation.html[scripted
metric aggregations] on your data. These aggregations are flexible and make
it possible to perform very complex processing. Let's use scripted metrics to
identify suspicious client IPs in the web log
{kibana-ref}/add-sample-data.html[{kib} sample dataset].

The example below transforms web log data into an entity-centric index where the
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
entity is `clientip`.

[source,js]
----------------------------------
POST _data_frame/transforms/_preview
{
"source": {
"index": "kibana_sample_data_logs",
"query": { <1>
"range" : {
"timestamp" : {
"gte" : "now-30d/d"
}
}
}
},
"dest" : {
"index" : "sample_weblogs_by_clientip"
},
"pivot": {
"group_by": { <2>
"clientip": { "terms": { "field": "clientip" } }
},
"aggregations": {
"url_dc": { "cardinality": { "field": "url.keyword" }},
"bytes_sum": { "sum": { "field": "bytes" }},
"geo.src_dc": { "cardinality": { "field": "geo.src" }},
"agent_dc": { "cardinality": { "field": "agent.keyword" }},
"geo.dest_dc": { "cardinality": { "field": "geo.dest" }},
"responses.total": { "value_count": { "field": "timestamp" }},
"responses.counts": { <3>
"scripted_metric": {
"init_script": "state.responses = ['error':0L,'success':0L,'other':0L]",
"map_script": """
def code = doc['response.keyword'].value;
if (code.startsWith('5') || code.startsWith('4')) {
state.responses.error += 1 ;
} else if(code.startsWith('2')) {
state.responses.success += 1;
} else {
state.responses.other += 1;
}
""",
"combine_script": "state.responses",
"reduce_script": """
def counts = ['error': 0L, 'success': 0L, 'other': 0L];
for (responses in states) {
counts.error += responses['error'];
counts.success += responses['success'];
counts.other += responses['other'];
}
return counts;
"""
}
},
"timestamp.min": { "min": { "field": "timestamp" }},
"timestamp.max": { "max": { "field": "timestamp" }},
"timestamp.duration_ms": { <4>
"bucket_script": {
"buckets_path": {
"min_time": "timestamp.min.value",
"max_time": "timestamp.max.value"
},
"script": "(params.max_time - params.min_time)"
}
}
}
}
}
----------------------------------
// CONSOLE
// TEST[skip:SETUP]

<1> This range query limits the transform to documents that are within the
last 30 days at the point in time the {dataframe-transform} is started.
szabosteve marked this conversation as resolved.
Show resolved Hide resolved
<2> The data is grouped by the `clientip` field.
<3> This `scripted_metric` performs a distributed operation on the web log data
to count specific types of HTTP responses (error, success, and other).
<4> This `bucket_script` calculates the duration of the `clientip` access based
on the results of the aggregation.

In this example, the API returns the following response (note that this example
contains the response partially and shows only the first document object):

[source,js]
----------------------------------
{
"preview" : [
{
"geo" : {
"src_dc" : 12.0,
"dest_dc" : 9.0
},
"clientip" : "0.72.176.46",
"agent_dc" : 3.0,
"responses" : {
"total" : 14.0,
"counts" : {
"other" : 0,
"success" : 14,
"error" : 0
}
},
"bytes_sum" : 74808.0,
"timestamp" : {
"duration_ms" : 4.919943239E9,
"min" : "2019-06-17T07:51:57.333Z",
"max" : "2019-08-13T06:31:00.572Z"
},
"url_dc" : 11.0
},
...
}
----------------------------------
// CONSOLE
// TEST[skip:SETUP]

This {dataframe-transform} makes it easier to answer questions such as:
szabosteve marked this conversation as resolved.
Show resolved Hide resolved

* Which client IPs are transferring the most amounts of data?
(Based on the `"bytes_sum": { "sum": { "field": "bytes" }},` aggregation.)
szabosteve marked this conversation as resolved.
Show resolved Hide resolved

* Which client IPs are interacting with a high number of different URLs?
(Based on the `"url_dc": { "cardinality": { "field": "url.keyword" }},`
aggregation.)

* Which client IPs have high error rates?
(Based on the `scripted_metric` aggregation.)

* Which client IPs are interacting with a high number of destination countries?
(Based on the `"geo.dest_dc": { "cardinality": { "field": "geo.dest" }},`
aggregation.)
3 changes: 2 additions & 1 deletion docs/en/stack/data-frames/index.asciidoc
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
include::dataframes.asciidoc[]
include::ecommerce-example.asciidoc[]
include::api-quickref.asciidoc[]
include::api-quickref.asciidoc[]
include::dataframe-examples.asciidoc[]