Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server side caching implementation for the Index Pattern api #5575

Closed
wants to merge 69 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
a05153e
organizing thoughts
Bargs Oct 27, 2015
fcef3a3
can has plugin
Bargs Oct 29, 2015
887b498
Add GET endpoints for index-patterns and specific index-pattern
Bargs Oct 29, 2015
5c0b187
Add POST and DELETE
Bargs Oct 29, 2015
7ff9b62
add PUT endpoint for updating index patterns
Bargs Oct 29, 2015
f51ea0b
adding broken tests
Bargs Oct 29, 2015
949a0b7
Handle common elasticsearch errors and return the appropriate status …
Bargs Oct 30, 2015
584a7a2
make the tests aware of their async nature
Bargs Nov 3, 2015
fe8bf0a
Starting work on index template creation
Bargs Nov 5, 2015
f0f2b41
Accept index template mappings when POSTing index pattern
Bargs Nov 10, 2015
55b410c
Roll back index pattern creation if the template creation fails for s…
Bargs Nov 11, 2015
f7ee8d8
Use template string for readability
Bargs Nov 11, 2015
3a4d8e5
only create an index template if the client provided mappings and the…
Bargs Nov 12, 2015
2b75af2
account for deprecated wildcard syntax
Bargs Nov 12, 2015
42b1287
Index template names must be lower case
Bargs Nov 12, 2015
da912b1
delete unit tests that will be replaced by functional tests
Bargs Nov 12, 2015
ee7dab3
don't allow 0 matches to return true for wildcard index names
Bargs Nov 12, 2015
8fd023d
Don't allow updates to mappings
Bargs Nov 12, 2015
c0edb9f
return mappings from template with index pattern
Bargs Nov 12, 2015
4c6036e
GET mostly working, still need to fix mappings if retreived from index
Bargs Nov 13, 2015
cdbe70d
Pull field mappings out of their nested structure
Bargs Nov 13, 2015
229deb1
refactor get mappings into a library function
Bargs Nov 13, 2015
15d62f0
GET all index-patterns now pulls back pattern with field mappings
Bargs Nov 13, 2015
10b3e78
Created helper function for translating between template and pattern …
Bargs Nov 14, 2015
3242c72
DELETE will now delete both index pattern and template
Bargs Nov 14, 2015
afe1d60
support GETting old index patterns in the new format
Bargs Nov 14, 2015
1da5b85
First API test
Bargs Nov 17, 2015
aff348c
disable xsrf protection when running api tests
Bargs Nov 17, 2015
14b2910
test for missing payload with POST
Bargs Nov 17, 2015
664f8e5
return from handler in empty payload check to prevent error from bein…
Bargs Nov 17, 2015
6e23729
Added Joi schema for index pattern resource and tests to make sure it…
Bargs Nov 18, 2015
c50fb28
Pass the original ES error message back with the correct status code
Bargs Nov 18, 2015
05946f6
adding a bunch of tests for the POST index-patterns endpoints
Bargs Nov 19, 2015
a0b1356
Refactor tests into modules
Bargs Nov 19, 2015
cdaaae1
Ensure that field mappings are given a type
Bargs Nov 19, 2015
af08660
fix function name
Bargs Nov 19, 2015
c728086
Test PUTting index patterns
Bargs Nov 19, 2015
442a2e6
Don't allow updates to title when PUTting index-patterns
Bargs Nov 19, 2015
58d3523
Test 400 responds for invalid PUT payloads
Bargs Nov 19, 2015
eabdfbe
test 404 response when PUTting to a non-existent document
Bargs Nov 19, 2015
e873de2
Test DELETE of index-patterns
Bargs Nov 19, 2015
bf9c996
Test GETting all index patterns with creation of test data before sui…
Bargs Nov 19, 2015
d0948e9
test GETting an index-pattern by ID
Bargs Nov 19, 2015
208bf03
use alternative lodash syntax for readability
Bargs Nov 19, 2015
95f34c2
adhear to new API naming conventions
Bargs Nov 19, 2015
6d6cc69
Move the API into the kibana plugin since index patterns are a core p…
Bargs Nov 19, 2015
f45556a
[] synatx doesn't mean anything to elasticsearch so it doesn't make s…
Bargs Nov 19, 2015
73d02e9
Converted GET index patterns over to using callWithRequest
Bargs Nov 19, 2015
34d95e4
Switch GET by ID to use callWithRequest
Bargs Nov 19, 2015
ba9739d
Convert POST index patterns to using callWithRequest
Bargs Nov 19, 2015
f360144
Convert PUT and DELETE index patterns to callWithRequest
Bargs Nov 19, 2015
d1a9294
fix caps in require statement
Bargs Nov 20, 2015
37af4f5
Give API tests their own intern config
Bargs Nov 20, 2015
db17d7c
Fix require for casesensitive systems
Bargs Nov 20, 2015
df78127
refactor index pattern endpoints into separate modules
Bargs Nov 24, 2015
c24530b
tests for handleESError util funtion
Bargs Nov 24, 2015
a3ee30b
Use snake_case in API payloads
Bargs Nov 26, 2015
4f07e26
add tests and documentation for stitch_pattern_and_mappings util func…
Bargs Dec 1, 2015
f4e121b
adhearing to styleguide
Bargs Dec 1, 2015
fad2bb9
refactor getMappings so it doesn't require the hapi request object to…
Bargs Dec 2, 2015
cb08042
Modify getMappings to use a bound version of callWithRequest to make …
Bargs Dec 3, 2015
4fa4b77
Refactor routes to use Hapi route options to setup request payload va…
Bargs Dec 3, 2015
91bade9
add tests and argument validation for converPatternAndTemplateName
Bargs Dec 3, 2015
ed16071
tests for removeDeprecatedFieldProps
Bargs Dec 3, 2015
6f90744
test deep and strict equality
Bargs Dec 3, 2015
1a4d13d
Add field mapping conflict detection
Bargs Dec 4, 2015
692dcc3
no need to check for array now that removeDeprecatedFieldProps can ac…
Bargs Dec 4, 2015
1df4153
Make PUT return 'success' to match POST
Bargs Dec 4, 2015
af497b1
Index api caching proof of concept
Bargs Dec 4, 2015
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@
"bootstrap": "3.3.5",
"brace": "0.5.1",
"bunyan": "1.4.0",
"catbox": "7.0.0",
"commander": "2.8.1",
"css-loader": "0.17.0",
"d3": "3.5.6",
Expand Down Expand Up @@ -174,6 +175,7 @@
"simple-git": "1.8.0",
"sinon": "1.17.2",
"source-map": "0.4.4",
"supertest-as-promised": "2.0.2",
"wreck": "6.2.0"
},
"engines": {
Expand Down
218 changes: 218 additions & 0 deletions src/fixtures/field_mapping_multi_index.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
{
"logstash-2015.12.03": {
"mappings": {
"_default_": {
"agent": {
"full_name": "agent",
"mapping": {
"agent": {
"index": "analyzed",
"type": "string",
"doc_values": false
}
}
},
"@timestamp": {
"full_name": "@timestamp",
"mapping": {
"@timestamp": {
"index": "not_analyzed",
"type": "date",
"doc_values": true
}
}
},
"bytes": {
"full_name": "bytes",
"mapping": {
"bytes": {
"index": "not_analyzed",
"type": "number",
"doc_values": true
}
}
},
"ip": {
"full_name": "ip",
"mapping": {
"ip": {
"index": "not_analyzed",
"type": "ip",
"doc_values": true
}
}
},
"geo.coordinates": {
"full_name": "geo.coordinates",
"mapping": {
"coordinates": {
"index": "not_analyzed",
"type": "geo_point",
"doc_values": false
}
}
}
},
"apache": {
"agent": {
"full_name": "agent",
"mapping": {
"agent": {
"index": "analyzed",
"type": "string",
"doc_values": false
}
}
},
"@timestamp": {
"full_name": "@timestamp",
"mapping": {
"@timestamp": {
"index": "not_analyzed",
"type": "date",
"doc_values": true
}
}
},
"bytes": {
"full_name": "bytes",
"mapping": {
"bytes": {
"index": "not_analyzed",
"type": "number",
"doc_values": true
}
}
},
"ip": {
"full_name": "ip",
"mapping": {
"ip": {
"index": "not_analyzed",
"type": "ip",
"doc_values": true
}
}
},
"geo.coordinates": {
"full_name": "geo.coordinates",
"mapping": {
"coordinates": {
"index": "not_analyzed",
"type": "geo_point",
"doc_values": false
}
}
}
}
}
},
"logstash-2015.12.04": {
"mappings": {
"_default_": {
"agent": {
"full_name": "agent",
"mapping": {
"agent": {
"index": "analyzed",
"type": "string",
"doc_values": false
}
}
},
"@timestamp": {
"full_name": "@timestamp",
"mapping": {
"@timestamp": {
"index": "not_analyzed",
"type": "date",
"doc_values": true
}
}
},
"bytes": {
"full_name": "bytes",
"mapping": {
"bytes": {
"index": "not_analyzed",
"type": "number",
"doc_values": true
}
}
},
"ip": {
"full_name": "ip",
"mapping": {
"ip": {
"index": "not_analyzed",
"type": "ip",
"doc_values": true
}
}
},
"geo.coordinates": {
"full_name": "geo.coordinates",
"mapping": {
"coordinates": {
"index": "not_analyzed",
"type": "geo_point",
"doc_values": false
}
}
}
},
"apache": {
"agent": {
"full_name": "agent",
"mapping": {
"agent": {
"index": "analyzed",
"type": "string",
"doc_values": false
}
}
},
"@timestamp": {
"full_name": "@timestamp",
"mapping": {
"@timestamp": {
"index": "not_analyzed",
"type": "date",
"doc_values": true
}
}
},
"bytes": {
"full_name": "bytes",
"mapping": {
"bytes": {
"index": "not_analyzed",
"type": "number",
"doc_values": true
}
}
},
"ip": {
"full_name": "ip",
"mapping": {
"ip": {
"index": "not_analyzed",
"type": "ip",
"doc_values": true
}
}
},
"geo.coordinates": {
"full_name": "geo.coordinates",
"mapping": {
"coordinates": {
"index": "not_analyzed",
"type": "geo_point",
"doc_values": false
}
}
}
}
}
}
}
26 changes: 26 additions & 0 deletions src/fixtures/index_pattern_with_mappings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"title": "logstash-*",
"timeFieldName": "@timestamp",
"fields": [
{
"name": "geo.coordinates",
"count": 0,
"scripted": false,
"mapping": {
"index": "not_analyzed",
"type": "geo_point",
"doc_values": false
}
},
{
"name": "ip",
"count": 2,
"scripted": false,
"mapping": {
"index": "not_analyzed",
"type": "ip",
"doc_values": true
}
}
]
}
39 changes: 39 additions & 0 deletions src/fixtures/index_template.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"kibana-logstash-*": {
"order": 0,
"template": "logstash-*",
"settings": {},
"mappings": {
"_default_": {
"properties": {
"agent": {
"index": "analyzed",
"type": "string",
"doc_values": false
},
"@timestamp": {
"index": "not_analyzed",
"type": "date",
"doc_values": true
},
"bytes": {
"index": "not_analyzed",
"type": "number",
"doc_values": true
},
"ip": {
"index": "not_analyzed",
"type": "ip",
"doc_values": true
},
"geo.coordinates": {
"index": "not_analyzed",
"type": "geo_point",
"doc_values": false
}
}
}
},
"aliases": {}
}
}
6 changes: 6 additions & 0 deletions src/plugins/kibana/index.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
const indexPatterns = require('./server/routes/api/index_patterns');

module.exports = function (kibana) {
return new kibana.Plugin({

Expand Down Expand Up @@ -42,6 +44,10 @@ module.exports = function (kibana) {
};
}
}
},

init: function (server, options) {
indexPatterns(server);
}
});

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
const {templateToPattern, patternToTemplate} = require('../convert_pattern_and_template_name');
const expect = require('expect.js');

describe('convertPatternAndTemplateName', function () {

describe('templateToPattern', function () {

it('should convert an index template\'s name to its matching index pattern\'s title', function () {
expect(templateToPattern('kibana-logstash-*')).to.be('logstash-*');
});

it('should throw an error if the template name isn\'t a valid kibana namespaced name', function () {
expect(templateToPattern).withArgs('logstash-*').to.throwException('not a valid kibana namespaced template name');
expect(templateToPattern).withArgs('').to.throwException(/not a valid kibana namespaced template name/);
});

});

describe('patternToTemplate', function () {

it('should convert an index pattern\'s title to its matching index template\'s name', function () {
expect(patternToTemplate('logstash-*')).to.be('kibana-logstash-*');
});

it('should throw an error if the pattern is empty', function () {
expect(patternToTemplate).withArgs('').to.throwException(/pattern must not be empty/);
});

});

});
Loading