Support for number, date and IP range data types #76971

wylieconlon · 2020-09-08T18:53:14Z

Elasticsearch range types, like integer_range, date_range, or ip_range, are now supported in only the following:

KQL filtering
Filtering using filter pills
Adding positive & negative filters from Discover
Visualize histograms
Visualize cardinality
Lens unique count
Lens field stats (no data is shown)

Filtering support:

Support in esaggs:

Closes #12751

Checklist

Unit or functional tests were updated or added to match the most common scenarios

joealex · 2020-09-22T22:15:59Z

What is the plan for ip_range support in Kibana. Without the support in Discover (search, filter, interesting fields, tables), Visualizations the power of this field cannot be used. Also the documentation is slightly confusing on field type ip_range and the range aggregation for field type ip

elasticmachine · 2021-01-29T23:45:29Z

Pinging @elastic/kibana-app-services (Team:AppServices)

wylieconlon · 2021-01-30T00:04:43Z

I think this PR actually needs to go back into draft mode, sorry for the noise. While it does kind-of-work, I would like to test all the apps that are affected by this. For example, adding a filter from Discover causes an invalid query.

spalger · 2021-02-03T18:19:19Z

jenkins test this

flash1293

Mostly checked out Lens, this works pretty well. Do we have an issue already for adding the ability to do number histograms on number ranges?

I noticed a bug based on this, opened a separate issue because it's not directly related: #93433

flash1293 · 2021-03-03T13:25:06Z

x-pack/plugins/lens/public/indexpattern_datasource/operations/definitions/ranges/ranges.tsx

@@ -127,6 +127,7 @@ export const rangeOperation: OperationDefinition<RangeIndexPatternColumn, 'field
    );
  },
  onFieldChange: (oldColumn, field) => {
+    // TODO: Handle switch with number_range fields


flash1293 · 2021-03-03T14:55:22Z

test/functional/apps/visualize/_histogram_request_start.ts

+      it('should use provided value when number of generated buckets is less than histogram:maxBars', async function () {
+        log.debug('Field = machine.ram_range');
+        await PageObjects.visEditor.selectField('machine.ram_range');
+        await PageObjects.common.sleep(1000); // fix this


Maybe we can retry the assertion in the next line instead of sleeping?

Done. This pattern is used throughout the file but I don't need it here.

wylieconlon · 2021-03-03T16:27:31Z

@flash1293 yes, I've updated the PR description to remove the Lens histogram changes, since we no longer support fixed interval histograms.

flash1293

Tested around and this looks good to me for the Lens integration.

One thing to consider:
With partially unbounded ranges and histograms it's super easy to run very expensive queries that eventually run into OOM. E.g if there's a date range which starts at 2020-02-03 without an end date and you are doing a date histogram on them, it will eventually cause an OOM error on Elasticsearch (probably because it never stops creating buckets). Not sure how we can help with this from Kibana side though. Maybe you have an idea?

wylieconlon · 2021-03-04T15:45:36Z

@flash1293 Does that happen when issuing the queries from Kibana, or just in dev tools? I think that the Kibana query would not do that because of the time filter + custom extents that are set on date histograms.

Either way, an OOM in Elasticsearch sounds like a bug. I found this issue which appears to have closed the bug: elastic/elasticsearch#50109

flash1293 · 2021-03-04T15:47:52Z

@wylieconlon interestingly it happened to me in Kibana as well. Steps to reproduce:

Run these requests

PUT abc
{
  "mappings": {
"properties": {
  "number_range": {
    "type": "integer_range"
  },
  "date_range": {
    "type": "date_range"
  }
}
  }
}

POST abc/_doc
{
  "number_range": {
    "gte": 12,
    "lte": 50
  },
  "date_range": {
    "gte": "2021-03-02",
    "lte": "2021-03-05"
  }
}

POST abc/_doc
{
  "number_range": {
    "gte": 30,
    "lte": 70
  },
  "date_range": {
    "gte": "2021-02-25",
    "lte": "2021-02-28"
  }
}

POST abc/_doc
{
  "number_range": {
    "gte": 60,
    "lte": 80
  },
  "date_range": {
    "gte": "2021-02-27",
    "lte": "2021-03-24"
  }
}


POST abc/_doc
{
  "number_range": {
    "gte": 60,
    "lte": 80
  },
  "date_range": {
    "gte": "2021-02-28"
  }
}

Configure abc index pattern
Go to Lens and use date_range on the x axis
Loads for a while, then the request fails and Elasticsearch logs OOMs

flash1293 · 2021-03-04T15:50:32Z

@wylieconlon The issue you sent fixes this by adding a new parameter: hard_bounds (elastic/elasticsearch#59175). Maybe we should take care of adding that to our requests before rolling out this feature in Kibana to make it harder to run into super expensive queries? I don't have a strong opinion on it though, it would be fine as a follow-up for me as well.

wylieconlon · 2021-03-04T15:55:54Z

@flash1293 Okay, I read the docs on hard_bounds. We can set it automatically on date_histograms and I will add that, but we can't set it automatically on histogram aggregations. We would have to ask users to enter the hard bounds as an advanced option. I think hard_bounds only makes sense for range data and I could add to the form here.

If possible, I'd like to do this in a separate PR. This PR has been going on for way too long.

flash1293

Sure, let's move it into a separate issue. LGTM otherwise

ppisljar

code LGTM

kibanamachine · 2021-03-08T20:49:33Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`data`	217.8KB	218.0KB	+205.0B
`lens`	919.9KB	920.6KB	+693.0B
`visDefaultEditor`	241.5KB	241.6KB	+120.0B
`visTypeTimeseries`	1.7MB	1.7MB	-2.0B
total			+1016.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`data`	827.1KB	828.9KB	+1.8KB
`dataEnhanced`	40.6KB	40.8KB	+195.0B
`kibanaReact`	127.2KB	127.3KB	+102.0B
total			+2.1KB

History

💚 Build #110914 succeeded 24d4b20
💔 Build #110869 failed 466dbb0
💔 Build #110856 failed 1e7357c
💛 Build #110809 was flaky 0582ce8
💔 Build #110791 failed e6e3dca

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Support for number, date and IP ranges * Update tests * Ranges don't work with range agg * Fix test case * Allow Discover to create range filters * Supports ranges in Visualize, KQL, remove Lens support * Fix test mappings * Bring back field cache to work around bug * Fix some tests * Fix test expectation * Respond to review comments * Fix type error * Remove added sample data * Fix api_docs * Fix test

Support for number, date and IP ranges

4d77749

wylieconlon added the Team:AppArch label Sep 8, 2020

wylieconlon added 3 commits September 8, 2020 15:30

Update tests

e2ad9de

Merge remote-tracking branch 'origin/master' into range-types

2c98e04

Ranges don't work with range agg

7e1fefb

lukeelmers added the Feature:Aggregations Aggregation infrastructure (AggConfig, esaggs, ...) label Sep 16, 2020

wylieconlon requested a review from lukeelmers November 24, 2020 23:06

wylieconlon mentioned this pull request Dec 31, 2020

[Lens] Add specific IP and Range/Interval sorting to datatable #87006

Merged

1 task

wylieconlon added 2 commits January 29, 2021 18:02

Merge remote-tracking branch 'origin/master' into range-types

f8c1478

Fix test case

43cf942

wylieconlon marked this pull request as ready for review January 29, 2021 23:45

wylieconlon requested a review from a team January 29, 2021 23:45

wylieconlon requested review from a team as code owners January 29, 2021 23:45

wylieconlon added release_note:enhancement v7.12.0 v8.0.0 labels Jan 29, 2021

wylieconlon changed the title ~~Support for number, date and IP ranges~~ Support for number, date and IP range data types Jan 29, 2021

wylieconlon marked this pull request as draft January 30, 2021 00:04

wylieconlon added 5 commits February 1, 2021 15:16

Merge remote-tracking branch 'origin/master' into range-types

4a3d312

Allow Discover to create range filters

5aff4e1

Merge remote-tracking branch 'origin/master' into range-types

3b08bbb

Supports ranges in Visualize, KQL, remove Lens support

9330884

Fix test mappings

d1e60eb

flash1293 reviewed Mar 3, 2021

View reviewed changes

Respond to review comments

e6e3dca

wylieconlon added 6 commits March 3, 2021 12:03

Fix type error

0582ce8

Merge remote-tracking branch 'origin/master' into range-types

3196f5b

Remove added sample data

1e7357c

Fix api_docs

466dbb0

Merge remote-tracking branch 'origin/master' into range-types

4a44b9d

Fix test

24d4b20

wylieconlon requested a review from ppisljar March 3, 2021 23:54

flash1293 reviewed Mar 4, 2021

View reviewed changes

wylieconlon requested a review from streamich March 4, 2021 15:43

flash1293 approved these changes Mar 4, 2021

View reviewed changes

wylieconlon mentioned this pull request Mar 4, 2021

[Aggs] Apply hard_bounds to prevent ES circuit breaker when dealing with open-ended ranges #93622

Open

ppisljar approved these changes Mar 8, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into range-types

c2d965c

wylieconlon merged commit c33987d into elastic:master Mar 8, 2021

wylieconlon deleted the range-types branch March 8, 2021 20:50

wylieconlon mentioned this pull request Mar 8, 2021

[7.x] Support for number, date and IP range data types (#76971) #94001

Merged

bhavyarm mentioned this pull request Oct 7, 2021

Replaces x-pack/test/functional/es_archives/logstash_functional with test/functional/fixtures/es_archiver/logstash_functional in test files #114189

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for number, date and IP range data types #76971

Support for number, date and IP range data types #76971

wylieconlon commented Sep 8, 2020 •

edited

Loading

joealex commented Sep 22, 2020

elasticmachine commented Jan 29, 2021

wylieconlon commented Jan 30, 2021

spalger commented Feb 3, 2021

flash1293 left a comment

flash1293 Mar 3, 2021

flash1293 Mar 3, 2021

wylieconlon Mar 3, 2021

wylieconlon commented Mar 3, 2021

flash1293 left a comment •

edited

Loading

wylieconlon commented Mar 4, 2021 •

edited

Loading

flash1293 commented Mar 4, 2021

flash1293 commented Mar 4, 2021 •

edited

Loading

wylieconlon commented Mar 4, 2021 •

edited

Loading

flash1293 left a comment

ppisljar left a comment

kibanamachine commented Mar 8, 2021

Support for number, date and IP range data types #76971

Support for number, date and IP range data types #76971

Conversation

wylieconlon commented Sep 8, 2020 • edited Loading

Checklist

joealex commented Sep 22, 2020

elasticmachine commented Jan 29, 2021

wylieconlon commented Jan 30, 2021

spalger commented Feb 3, 2021

flash1293 left a comment

Choose a reason for hiding this comment

flash1293 Mar 3, 2021

Choose a reason for hiding this comment

flash1293 Mar 3, 2021

Choose a reason for hiding this comment

wylieconlon Mar 3, 2021

Choose a reason for hiding this comment

wylieconlon commented Mar 3, 2021

flash1293 left a comment • edited Loading

Choose a reason for hiding this comment

wylieconlon commented Mar 4, 2021 • edited Loading

flash1293 commented Mar 4, 2021

flash1293 commented Mar 4, 2021 • edited Loading

wylieconlon commented Mar 4, 2021 • edited Loading

flash1293 left a comment

Choose a reason for hiding this comment

ppisljar left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 8, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

History

wylieconlon commented Sep 8, 2020 •

edited

Loading

flash1293 left a comment •

edited

Loading

wylieconlon commented Mar 4, 2021 •

edited

Loading

flash1293 commented Mar 4, 2021 •

edited

Loading

wylieconlon commented Mar 4, 2021 •

edited

Loading