-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Fleet] experimental toggles for doc-value-only (#149131)
## Summary Closes #144357 WIP. Review can be started, but still requires a lot of testing and fixing the issue below. How to test locally: - Turn on `experimentalDataStreamSettings` feature flag - Go to Add integration, System integration - On the first data stream, turn on the Doc value only switches, Save - The mapping changes are visible under Stack Management / Index Management / Component Templates e.g. `logs-system.auth@package` - The numeric switch sets `index:false` on all numeric field mappings (long, double, etc.) - The other switch sets `index:false` on all other field type mappings that support it (keyword, ip, date, etc.) - The new mappings will take effect after rollover <img width="475" alt="image" src="https://user-images.githubusercontent.com/90178898/213206641-13ead2fc-f079-407c-9c0e-c58f99dd4903.png"> <img width="1037" alt="image" src="https://user-images.githubusercontent.com/90178898/213495546-9962c458-590b-4787-bf2d-9f19abea3f67.png"> What works: - When turning the new doc-value-only numeric and other checkboxes on or off, the corresponding mapping changes are done in the component template - The logic also reads the package spec's template and preserves the `index:false` values regardless of the switch (tested manually by setting `@timestamp` field to `index:false` in the template, there is also the `original` field in `logs-system.auth@package` stream that is set to `index:false` in the package by default. ``` "original": { "index": false, "doc_values": false, "type": "keyword" }, ``` Pending: - Add/update unit and integration tests to verify the mapping change logic - DONE - Manual testing (turning the switches on/off, create/update package policy, upgrade package) - DONE - Clarify TODOs in the code about the supported types - DONE - Hitting an issue when turning on `doc-value-only` for "other" types (keyword, date, etc.). Could be that one of the fields doesn't support `index:false` setting. Didn't experience this when turning on only the numeric types. - FIXED ``` illegal_argument_exception: [illegal_argument_exception] Reason: updating component template [logs-system.auth@package] results in invalid composable template [logs-system.auth] after templates are merged ``` EDIT: found the root cause of this: `Caused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is not indexed` ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Kibana Machine <[email protected]>
- Loading branch information
1 parent
e2824c3
commit 179b36f
Showing
12 changed files
with
510 additions
and
55 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
70 changes: 70 additions & 0 deletions
70
x-pack/plugins/fleet/server/services/experimental_datastream_features_helper.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import type { | ||
| MappingProperty, | ||
| PropertyName, | ||
| } from '@elastic/elasticsearch/lib/api/typesWithBodyKey'; | ||
|
|
||
| import type { ExperimentalDataStreamFeature } from '../../common/types'; | ||
|
|
||
| export const forEachMappings = ( | ||
| mappingProperties: Record<PropertyName, MappingProperty>, | ||
| process: (prop: MappingProperty, name: string) => void | ||
| ) => { | ||
| Object.keys(mappingProperties).forEach((mapping) => { | ||
| const property = mappingProperties[mapping] as any; | ||
| if (property.properties) { | ||
| forEachMappings(property.properties, process); | ||
| } else { | ||
| process(property, mapping); | ||
| } | ||
| }); | ||
| }; | ||
|
|
||
| export const applyDocOnlyValueToMapping = ( | ||
| mappingProp: MappingProperty, | ||
| name: string, | ||
| featureMap: ExperimentalDataStreamFeature, | ||
| isDocValueOnlyNumericChanged: boolean, | ||
| isDocValueOnlyOtherChanged: boolean | ||
| ) => { | ||
| const mapping = mappingProp as any; | ||
|
|
||
| const numericTypes = [ | ||
| 'long', | ||
| 'integer', | ||
| 'short', | ||
| 'byte', | ||
| 'double', | ||
| 'float', | ||
| 'half_float', | ||
| 'scaled_float', | ||
| 'unsigned_long', | ||
| ]; | ||
| if (isDocValueOnlyNumericChanged && numericTypes.includes(mapping.type ?? '')) { | ||
| updateMapping(mapping, featureMap.features.doc_value_only_numeric); | ||
| } | ||
|
|
||
| const otherSupportedTypes = ['date', 'date_nanos', 'boolean', 'ip', 'geo_point', 'keyword']; | ||
| if ( | ||
| isDocValueOnlyOtherChanged && | ||
| name !== '@timestamp' && | ||
| otherSupportedTypes.includes(mapping.type ?? '') | ||
| ) { | ||
| updateMapping(mapping, featureMap.features.doc_value_only_other); | ||
| } | ||
| }; | ||
|
|
||
| const updateMapping = (mapping: { index?: boolean }, featureValue: boolean) => { | ||
| if (featureValue === false && mapping.index === false) { | ||
| delete mapping.index; | ||
| } | ||
| if (featureValue && !mapping.index) { | ||
| mapping.index = false; | ||
| } | ||
| }; |
Oops, something went wrong.