-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* WIP: Adding new route for EQL Validation This is mostly boilerplate with some rough parameter definitions; the actual implementation of the validation is going to live in our validateEql function. A few tests are failing as the mocks haven't yet been implemented, I need to see the shape of the responses first. * Cherry-pick Marshall's EQL types * Implements actual EQL validation * Performs an EQL search * filters out non-parsing errors, and returns what remains in the response * Adds mocks for empty EQL responses (we don't yet have a need for mocked data, but we will when we unit-test validateEql) * Adds validation calls to the EQL form input * Adds EQL Validation response schema,mocks,tests * Adds frontend function to call our validation endpoint * Adds hook, useEqlValidation, to call the above function and return state * Adds labels/help text for EQL Query bar * EqlQueryBar consumes useEqlValidation and marks the field as invalid, but does not yet report errors. * Do not call the validation API if query is not present This causes a broader error that results in a 400 response; we can (and do) handle the case of a blank query in the form itself. * Remove EQL Help Text It doesn't add any information for the user, and it currently looks bad when combined with validation errors. * Flesh out and use our popover for displaying validation errors * Fixes issue where old errors were persisted after the user had made modifications * Include verification_exception errors as validation errors These include errors related to index fields and mappings. * Generalize our validation helpers We're concerned with validation errors; the source of those errors is an implementation detail of these functions. * Move error popover and EQL reference link to footer This more closely resembles the new Eui Markdown editor, which places errors and doc links in a footer. * Fix jest tests following additional prop * Add icon for EQL Rule card * Fixes existing EqlQueryBar tests These were broken by our use of useAppToasts and the EUI theme. * Add unit tests around error rendering on EQL Query Bar * Add tests for ErrorPopover * Remove unused schema type Decode doesn't do any additional processing, so we can use t.TypeOf here (the default for buildRouteValidation). * Remove duplicated header * Use ignore parameter to prevent EQL validations from logging errors Without `ignore: [400]` the ES client will log errors and then throw them. We can catch the error, but the log is undesirable. This updates the query to use the ignore parameter, along with updating the validation logic to work with the updated response. Adds some mocks and tests around these responses and helpers, since these will exist independent of the validation implementation. * Include mapping_exceptions during EQL query validation These include errors for inaccessible indexes, which should be useful to the rule writer in writing their EQL query. * Display toast messages for non-validation messages * fix type errors This type was renamed. * Do not request data in our validation request By not having the cluster retrieve/send any data, this should saves us a few CPU cycles. * Move EQL validation to an async form validator Rather than invoking a custom validation hook (useEqlValidation) at custom times (onBlur) in our EqlQueryBar component, we can instead move this functionality to a form validation function and have it be invoked automatically by our form when values change. However, because we still need to handle the validation messages slightly differently (place them in a popover as opposed to an EuiFormRow), we also need custom error retrieval in the form of getValidationResults. After much pain, it was determined that the default behavior of _.debounce does not work with async validator functions, as a debounced call will not "wait" for the eventual invocation but will instead return the most recently resolved value. This leads to stale validation results and terrible UX, so I wrote a custom function (debounceAsync) that behaves like we want/need; see tests for details. * Invalidate our query field when index patterns change Since EQL rules actually validate against the relevant indexes, changing said indexes should invalidate/revalidate the query. With the form lib, this is beautifully simple :) * Set a min-height on our EQL textarea * Remove unused prop from EqlQueryBar Index corresponds to the value from the index field; now that our EQL validation is performed by the form we have no need for it here. * Update EQL overview link to point to elasticsearch docs Adds an entry in our doclinks service, and uses that. * Remove unused prop from stale tests * Update docLinks documentation with new EQL link * Fix bug where saved query rules had no type selected on Edit * Wait for kibana requests to complete before moving between rule tabs With our new async validation, a user can quickly navigate away from the Definition tab before the validation has completed, resulting in the form being invalidated. Any subsequent user actions cause the form to correct itself, but until I can find a better solution here this really just gives the validation time to complete and sidesteps the issue.
- Loading branch information
Showing
41 changed files
with
1,075 additions
and
29 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
12 changes: 12 additions & 0 deletions
12
...s/security_solution/common/detection_engine/schemas/request/eql_validation_schema.mock.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
|
||
import { EqlValidationSchema } from './eql_validation_schema'; | ||
|
||
export const getEqlValidationSchemaMock = (): EqlValidationSchema => ({ | ||
index: ['index-123'], | ||
query: 'process where process.name == "regsvr32.exe"', | ||
}); |
59 changes: 59 additions & 0 deletions
59
...s/security_solution/common/detection_engine/schemas/request/eql_validation_schema.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
|
||
import { pipe } from 'fp-ts/lib/pipeable'; | ||
import { left } from 'fp-ts/lib/Either'; | ||
|
||
import { exactCheck } from '../../../exact_check'; | ||
import { foldLeftRight, getPaths } from '../../../test_utils'; | ||
import { eqlValidationSchema, EqlValidationSchema } from './eql_validation_schema'; | ||
import { getEqlValidationSchemaMock } from './eql_validation_schema.mock'; | ||
|
||
describe('EQL validation schema', () => { | ||
it('requires a value for index', () => { | ||
const payload = { | ||
...getEqlValidationSchemaMock(), | ||
index: undefined, | ||
}; | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
|
||
expect(getPaths(left(message.errors))).toEqual([ | ||
'Invalid value "undefined" supplied to "index"', | ||
]); | ||
expect(message.schema).toEqual({}); | ||
}); | ||
|
||
it('requires a value for query', () => { | ||
const payload = { | ||
...getEqlValidationSchemaMock(), | ||
query: undefined, | ||
}; | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
|
||
expect(getPaths(left(message.errors))).toEqual([ | ||
'Invalid value "undefined" supplied to "query"', | ||
]); | ||
expect(message.schema).toEqual({}); | ||
}); | ||
|
||
it('validates a payload with index and query', () => { | ||
const payload = getEqlValidationSchemaMock(); | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
const expected: EqlValidationSchema = { | ||
index: ['index-123'], | ||
query: 'process where process.name == "regsvr32.exe"', | ||
}; | ||
|
||
expect(getPaths(left(message.errors))).toEqual([]); | ||
expect(message.schema).toEqual(expected); | ||
}); | ||
}); |
18 changes: 18 additions & 0 deletions
18
...lugins/security_solution/common/detection_engine/schemas/request/eql_validation_schema.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
|
||
import * as t from 'io-ts'; | ||
|
||
import { index, query } from '../common/schemas'; | ||
|
||
export const eqlValidationSchema = t.exact( | ||
t.type({ | ||
index, | ||
query, | ||
}) | ||
); | ||
|
||
export type EqlValidationSchema = t.TypeOf<typeof eqlValidationSchema>; |
17 changes: 17 additions & 0 deletions
17
.../security_solution/common/detection_engine/schemas/response/eql_validation_schema.mock.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
|
||
import { EqlValidationSchema } from './eql_validation_schema'; | ||
|
||
export const getEqlValidationResponseMock = (): EqlValidationSchema => ({ | ||
valid: false, | ||
errors: ['line 3:52: token recognition error at: '], | ||
}); | ||
|
||
export const getValidEqlValidationResponseMock = (): EqlValidationSchema => ({ | ||
valid: true, | ||
errors: [], | ||
}); |
59 changes: 59 additions & 0 deletions
59
.../security_solution/common/detection_engine/schemas/response/eql_validation_schema.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
|
||
import { pipe } from 'fp-ts/lib/pipeable'; | ||
import { left } from 'fp-ts/lib/Either'; | ||
|
||
import { exactCheck } from '../../../exact_check'; | ||
import { foldLeftRight, getPaths } from '../../../test_utils'; | ||
import { getEqlValidationResponseMock } from './eql_validation_schema.mock'; | ||
import { eqlValidationSchema } from './eql_validation_schema'; | ||
|
||
describe('EQL validation response schema', () => { | ||
it('validates a typical response', () => { | ||
const payload = getEqlValidationResponseMock(); | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
|
||
expect(getPaths(left(message.errors))).toEqual([]); | ||
expect(message.schema).toEqual(getEqlValidationResponseMock()); | ||
}); | ||
|
||
it('invalidates a response with extra properties', () => { | ||
const payload = { ...getEqlValidationResponseMock(), extra: 'nope' }; | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
|
||
expect(getPaths(left(message.errors))).toEqual(['invalid keys "extra"']); | ||
expect(message.schema).toEqual({}); | ||
}); | ||
|
||
it('invalidates a response with missing properties', () => { | ||
const payload = { ...getEqlValidationResponseMock(), valid: undefined }; | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
|
||
expect(getPaths(left(message.errors))).toEqual([ | ||
'Invalid value "undefined" supplied to "valid"', | ||
]); | ||
expect(message.schema).toEqual({}); | ||
}); | ||
|
||
it('invalidates a response with properties of the wrong type', () => { | ||
const payload = { ...getEqlValidationResponseMock(), errors: 'should be an array' }; | ||
const decoded = eqlValidationSchema.decode(payload); | ||
const checked = exactCheck(payload, decoded); | ||
const message = pipe(checked, foldLeftRight); | ||
|
||
expect(getPaths(left(message.errors))).toEqual([ | ||
'Invalid value "should be an array" supplied to "errors"', | ||
]); | ||
expect(message.schema).toEqual({}); | ||
}); | ||
}); |
16 changes: 16 additions & 0 deletions
16
...ugins/security_solution/common/detection_engine/schemas/response/eql_validation_schema.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
/* | ||
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
* or more contributor license agreements. Licensed under the Elastic License; | ||
* you may not use this file except in compliance with the Elastic License. | ||
*/ | ||
|
||
import * as t from 'io-ts'; | ||
|
||
export const eqlValidationSchema = t.exact( | ||
t.type({ | ||
valid: t.boolean, | ||
errors: t.array(t.string), | ||
}) | ||
); | ||
|
||
export type EqlValidationSchema = t.TypeOf<typeof eqlValidationSchema>; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.