[Security Solution] New Data Quality dashboard Same family category (…

…#167480) # [Security Solution] New Data Quality dashboard `Same family` category This PR introduces a new `Same family` category to the [Data Quality dashboard](https://www.elastic.co/guide/en/security/current/data-quality-dash.html). Fields with the `same family` tag that were previously counted in the `Incompatible fields` category are now counted in the new `Same family` category, per the annotated screenshot below: ![same_family_before_after_annotated](https://github.com/elastic/kibana/assets/4459398/c7c5d496-03e5-4e06-b837-85f2f16ab885) _Above - Left: previously, fields with the `same family` tag were counted as `Incompatible fields`, Right: in this PR, those fields are counted in the new `Same family` category_ The annotations on the _Right_ side of the screenshot above highlight (in this example): - The total `Incompatible fields` count was reduced - Some patterns, like `auditbeat-*` have significant reductions - The `Incompatible fields` count was reduced by one field - The new `Same family` category has one field - The `agent.type` field moved from the `Incompatible fields` category to the `Same family` category ## Details ### The new `Same family` category Fields with mappings in the same [family](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html) have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics. The color of the `Same family` category is the same as the `Custom fields` category in badges and charts, per the screenshot below: ![same_family_color](https://github.com/elastic/kibana/assets/4459398/259b7e8e-3a52-482e-a419-edf715f8e462) _Above: The `Same family` tab's badge and chart legend color is the same as the `Custom fields` category_ ### The new `Same family` tab This PR introduces a new `Same family` tab, as shown in the screenshot below: ![same_family_tab](https://github.com/elastic/kibana/assets/4459398/f4da99cf-6fa4-4477-8a85-eec9d85c6787) _Above: The new `Same family` tab is selected_ In the screenshot above: - The callout includes a description of fields in the same family (moved from the `Incompatible fields` tab) - The `constant_keyword` text, yellow in previous versions (when it appered in the `Incompatible fields` tab), is blue - Only one action, `Copy to clipboard` is available in the `Same family` tab. The remaining text in this _Details_ section is the markdown copied to the clipboard for the example above: ### auditbeat-custom-index-1 | Result | Index | Docs | Incompatible fields | ILM Phase | Size | |--------|-------|------|---------------------|-----------|------| | ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 13.1KB | ### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10` #### 1 Same family field mapping This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match. Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics. #### Same family field mappings - auditbeat-custom-index-1 | Field | ECS mapping type (expected) | Index mapping type (actual) | |-------|-----------------------------|-----------------------------| | agent.type | `keyword` | `constant_keyword` `same family` | ## Desk testing 1) Navigate to `Dev Tools` > `Console` 2) Execute the queries below: <details> <summary>Queries to create the `auditbeat-custom-index-1` example in this PR description</summary> ``` DELETE auditbeat-custom-index-1 PUT auditbeat-custom-index-1 PUT auditbeat-custom-index-1/_mapping { "properties": { "@timestamp": { "type": "date" }, "agent.type": { "type": "constant_keyword" }, "event.category": { "type": "constant_keyword" } } } POST auditbeat-custom-index-1/_doc { "@timestamp": "2023-02-06T09:41:49.668Z", "host": { "name": "foo" }, "event": { "category": "an_invalid_category" }, "some.field": "this", "source": { "port": 90210, "ip": "10.1.2.3" } } POST auditbeat-custom-index-1/_doc { "@timestamp": "2023-02-06T09:42:22.123Z", "host": { "name": "bar" }, "event": { "category": "an_invalid_category" }, "some.field": "space", "source": { "port": 867, "ip": "10.9.8.7" } } ``` </details> 3) Navigate to `Security` > `Dashboards` > `Data Quality` 4) Expand the `auditbeat-custom-index-1` index **Expected results** - The sum of the category badge counts, `3 + 1 + 4 + 2`, equals the total number of fields in the `All fields` [`10`] category: - `Incompatible fields` [`3`] - `Same family` [`1`] - `Custom fields` [`4`] - `ECS compliant fields` [`2`] - The `Incompatible fields` callout title is `3 incompatible fields` - The `Incompatible fields` callout does NOT include a description of same family fields - The `Incompatible fields` tab displays two fields with incompatible mappings - `host.name` (`keyword` vs `text`) - `source.ip` (`ip` vs `text`) - The `Incompatible fields` tab displays one field with incompatible field values - `event.category` (`an_invalid_category`) 5) Click the `Summary` tab **Expected results** - The `Summary` tab is focused - The chart legend includes a `Same family` entry, with a count of `1` - The `Same family` tab's badge color is the same as the `Custom fields` category - The `Same family` chart legend color is the same as the `Custom fields` category 6) Click the `Same family` legend item **Expected results** - The `Same family` tab is focused - One field, `agent.type`, is displayed - The `constant_keyword` Index mapping type is blue - The `same family` badge is yellow 7) Click `Copy to clipboard` **Expected result** - The expected markdown is copied to the clipboard: ``` ### auditbeat-custom-index-1 | Result | Index | Docs | Incompatible fields | ILM Phase | Size | |--------|-------|------|---------------------|-----------|------| | ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 12.9KB | ### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10` #### 1 Same family field mapping This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match. Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics. #### Same family field mappings - auditbeat-custom-index-1 | Field | ECS mapping type (expected) | Index mapping type (actual) | |-------|-----------------------------|-----------------------------| | agent.type | `keyword` | `constant_keyword` `same family` | ```
elastic · Oct 2, 2023 · fe5abc9 · fe5abc9
1 parent 26e61c1
commit fe5abc9
Show file tree

Hide file tree

Showing 46 changed files with 1,492 additions and 160 deletions.
diff --git a/..._dashboard/impl/data_quality/compare_fields_table/get_common_table_columns/index.test.tsx b/..._dashboard/impl/data_quality/compare_fields_table/get_common_table_columns/index.test.tsx
@@ -122,8 +122,8 @@ describe('getCommonTableColumns', () => {
         );
       });
 
-      test('it renders the expected type as a warning', () => {
-        expect(screen.getByTestId('codeWarning')).toHaveTextContent(indexFieldType);
+      test('it renders the index field with a "success" style', () => {
+        expect(screen.getByTestId('codeSuccess')).toHaveTextContent(indexFieldType);
       });
 
       test('it renders the same family badge', () => {

diff --git a/...ality_dashboard/impl/data_quality/compare_fields_table/get_common_table_columns/index.tsx b/...ality_dashboard/impl/data_quality/compare_fields_table/get_common_table_columns/index.tsx
@@ -11,8 +11,9 @@ import React from 'react';
 
 import { SameFamily } from '../../data_quality_panel/same_family';
 import { EcsAllowedValues } from '../ecs_allowed_values';
+import { getIsInSameFamily } from '../../helpers';
 import { IndexInvalidValues } from '../index_invalid_values';
-import { CodeDanger, CodeSuccess, CodeWarning } from '../../styles';
+import { CodeDanger, CodeSuccess } from '../../styles';
 import * as i18n from '../translations';
 import type { AllowedValue, EnrichedFieldMetadata, UnallowedValueCount } from '../../types';
 
@@ -45,9 +46,9 @@ export const getCommonTableColumns = (): Array<
     name: i18n.INDEX_MAPPING_TYPE_ACTUAL,
     render: (_, x) =>
       x.type != null && x.indexFieldType !== x.type ? (
-        x.isInSameFamily ? (
+        getIsInSameFamily({ ecsExpectedType: x.type, type: x.indexFieldType }) ? (
           <div>
-            <CodeWarning data-test-subj="codeWarning">{x.indexFieldType}</CodeWarning>
+            <CodeSuccess data-test-subj="codeSuccess">{x.indexFieldType}</CodeSuccess>
             <SameFamily />
           </div>
         ) : (

diff --git a/.../data_quality/compare_fields_table/get_incompatible_mappings_table_columns/index.test.tsx b/.../data_quality/compare_fields_table/get_incompatible_mappings_table_columns/index.test.tsx
@@ -105,8 +105,8 @@ describe('getIncompatibleMappingsTableColumns', () => {
         );
       });
 
-      test('it renders the expected type as a warning', () => {
-        expect(screen.getByTestId('codeWarning')).toHaveTextContent(indexFieldType);
+      test('it renders the expected type with a "success" style', () => {
+        expect(screen.getByTestId('codeSuccess')).toHaveTextContent(indexFieldType);
       });
 
       test('it renders the same family badge', () => {

diff --git a/.../impl/data_quality/compare_fields_table/get_incompatible_mappings_table_columns/index.tsx b/.../impl/data_quality/compare_fields_table/get_incompatible_mappings_table_columns/index.tsx
@@ -9,7 +9,7 @@ import type { EuiTableFieldDataColumnType } from '@elastic/eui';
 import React from 'react';
 
 import { SameFamily } from '../../data_quality_panel/same_family';
-import { CodeDanger, CodeSuccess, CodeWarning } from '../../styles';
+import { CodeDanger, CodeSuccess } from '../../styles';
 import * as i18n from '../translations';
 import type { EnrichedFieldMetadata } from '../../types';
 
@@ -43,7 +43,7 @@ export const getIncompatibleMappingsTableColumns = (): Array<
     render: (indexFieldType: string, x) =>
       x.isInSameFamily ? (
         <div>
-          <CodeWarning data-test-subj="codeWarning">{indexFieldType}</CodeWarning>
+          <CodeSuccess data-test-subj="codeSuccess">{indexFieldType}</CodeSuccess>
           <SameFamily />
         </div>
       ) : (

diff --git a/..._quality/data_quality_panel/data_quality_summary/summary_actions/check_all/check_index.ts b/..._quality/data_quality_panel/data_quality_summary/summary_actions/check_all/check_index.ts
@@ -21,6 +21,7 @@ export const EMPTY_PARTITIONED_FIELD_METADATA: PartitionedFieldMetadata = {
   custom: [],
   ecsCompliant: [],
   incompatible: [],
+  sameFamily: [],
 };
 
 export async function checkIndex({

diff --git a/...a_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/helpers.test.ts b/...a_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/helpers.test.ts
@@ -71,6 +71,7 @@ describe('helpers', () => {
             type: 'date',
           },
         ],
+        sameFamily: [],
       };
 
       test('it returns a `PartitionedFieldMetadata` with an `incompatible` `@timestamp` when  `mappingsProperties` is undefined', () => {
@@ -301,7 +302,7 @@ describe('helpers', () => {
             ],
             hasEcsMetadata: true,
             isEcsCompliant: false,
-            isInSameFamily: true,
+            isInSameFamily: false,
           },
           {
             dashed_name: 'host-name',
@@ -620,7 +621,7 @@ describe('helpers', () => {
             ],
             hasEcsMetadata: true,
             isEcsCompliant: false,
-            isInSameFamily: true,
+            isInSameFamily: false,
           },
           {
             dashed_name: 'host-name',
@@ -657,6 +658,7 @@ describe('helpers', () => {
             isInSameFamily: false,
           },
         ],
+        sameFamily: [],
       });
     });
   });

diff --git a/...s_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/helpers.ts b/...s_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/helpers.ts
@@ -23,13 +23,15 @@ export const ALL_TAB_ID = 'allTab';
 export const ECS_COMPLIANT_TAB_ID = 'ecsCompliantTab';
 export const CUSTOM_TAB_ID = 'customTab';
 export const INCOMPATIBLE_TAB_ID = 'incompatibleTab';
+export const SAME_FAMILY_TAB_ID = 'sameFamilyTab';
 export const SUMMARY_TAB_ID = 'summaryTab';
 
 export const EMPTY_METADATA: PartitionedFieldMetadata = {
   all: [],
   ecsCompliant: [],
   custom: [],
   incompatible: [],
+  sameFamily: [],
 };
 
 export const getSortedPartitionedFieldMetadata = ({

diff --git a/...cs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/index.tsx b/...cs_data_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/index.tsx
@@ -227,6 +227,11 @@ const IndexPropertiesComponent: React.FC<Props> = ({
           ? partitionedFieldMetadata.incompatible.length
           : undefined;
 
+      const indexSameFamily: number | undefined =
+        error == null && partitionedFieldMetadata != null
+          ? partitionedFieldMetadata.sameFamily.length
+          : undefined;
+
       if (patternRollup != null) {
         const markdownComments =
           partitionedFieldMetadata != null
@@ -255,6 +260,7 @@ const IndexPropertiesComponent: React.FC<Props> = ({
               indexName,
               markdownComments,
               pattern,
+              sameFamily: indexSameFamily,
             },
           },
         });

diff --git a/..._dashboard/impl/data_quality/data_quality_panel/index_properties/markdown/helpers.test.ts b/..._dashboard/impl/data_quality/data_quality_panel/index_properties/markdown/helpers.test.ts
@@ -402,7 +402,7 @@ describe('helpers', () => {
   describe('getTabCountsMarkdownComment', () => {
     test('it returns a comment with the expected counts', () => {
       expect(getTabCountsMarkdownComment(mockPartitionedFieldMetadata)).toBe(
-        '### **Incompatible fields** `3` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n'
+        '### **Incompatible fields** `3` **Same family** `0` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `9`\n'
       );
     });
   });

diff --git a/...ality_dashboard/impl/data_quality/data_quality_panel/index_properties/markdown/helpers.ts b/...ality_dashboard/impl/data_quality/data_quality_panel/index_properties/markdown/helpers.ts
@@ -201,6 +201,8 @@ export const getTabCountsMarkdownComment = (
 ): string =>
   `### **${i18n.INCOMPATIBLE_FIELDS}** ${getCodeFormattedValue(
     `${partitionedFieldMetadata.incompatible.length}`
+  )} **${i18n.SAME_FAMILY}** ${getCodeFormattedValue(
+    `${partitionedFieldMetadata.sameFamily.length}`
   )} **${i18n.CUSTOM_FIELDS}** ${getCodeFormattedValue(
     `${partitionedFieldMetadata.custom.length}`
   )} **${i18n.ECS_COMPLIANT_FIELDS}** ${getCodeFormattedValue(

diff --git a/...a_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/translations.ts b/...a_quality_dashboard/impl/data_quality/data_quality_panel/index_properties/translations.ts
@@ -276,6 +276,22 @@ export const CUSTOM_CALLOUT = ({ fieldCount, version }: { fieldCount: number; ve
       '{fieldCount, plural, =1 {This field is not} other {These fields are not}} defined by the Elastic Common Schema (ECS), version {version}.',
   });
 
+export const SAME_FAMILY_CALLOUT = ({
+  fieldCount,
+  version,
+}: {
+  fieldCount: number;
+  version: string;
+}) =>
+  i18n.translate(
+    'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyCallout',
+    {
+      values: { fieldCount, version },
+      defaultMessage:
+        "{fieldCount, plural, =1 {This field is} other {These fields are}} defined by the Elastic Common Schema (ECS), version {version}, but {fieldCount, plural, =1 {its mapping type doesn't} other {their mapping types don't}} exactly match.",
+    }
+  );
+
 export const CUSTOM_CALLOUT_TITLE = (fieldCount: number) =>
   i18n.translate(
     'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.customCalloutTitle',
@@ -286,6 +302,16 @@ export const CUSTOM_CALLOUT_TITLE = (fieldCount: number) =>
     }
   );
 
+export const SAME_FAMILY_CALLOUT_TITLE = (fieldCount: number) =>
+  i18n.translate(
+    'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyCalloutTitle',
+    {
+      values: { fieldCount },
+      defaultMessage:
+        '{fieldCount} Same family {fieldCount, plural, =1 {field mapping} other {field mappings}}',
+    }
+  );
+
 export const CUSTOM_EMPTY = i18n.translate(
   'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.customEmptyContent',
   {
@@ -307,13 +333,7 @@ export const INCOMPATIBLE_FIELDS = i18n.translate(
   }
 );
 
-export const INCOMPATIBLE_CALLOUT = ({
-  fieldCount,
-  version,
-}: {
-  fieldCount: number;
-  version: string;
-}) =>
+export const INCOMPATIBLE_CALLOUT = (version: string) =>
   i18n.translate(
     'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout',
     {
@@ -323,34 +343,27 @@ export const INCOMPATIBLE_CALLOUT = ({
     }
   );
 
-export const INCOMPATIBLE_FIELDS_WITH = i18n.translate(
-  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.incompatibleFieldsWithLabel',
+export const FIELDS_WITH_MAPPINGS_SAME_FAMILY = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.fieldsWithMappingsSameFamilyLabel',
   {
     defaultMessage:
-      'Incompatible fields with mappings in the same family have exactly the same search behavior but may have different space usage or performance characteristics.',
+      'Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.',
   }
 );
 
-export const WHEN_AN_INCOMPATIBLE_FIELD = i18n.translate(
-  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAnIncompatibleFieldLabel',
+export const WHEN_A_FIELD_IS_INCOMPATIBLE = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCallout.whenAFieldIsIncompatibleLabel',
   {
-    defaultMessage: 'When an incompatible field is not in the same family:',
+    defaultMessage: 'When a field is incompatible:',
   }
 );
 
-export const INCOMPATIBLE_CALLOUT_TITLE = ({
-  fieldCount,
-  fieldsInSameFamily,
-}: {
-  fieldCount: number;
-  fieldsInSameFamily: number;
-}) =>
+export const INCOMPATIBLE_CALLOUT_TITLE = (fieldCount: number) =>
   i18n.translate(
     'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.incompatibleCalloutTitle',
     {
-      values: { fieldCount, fieldsInSameFamily },
-      defaultMessage:
-        '{fieldCount} incompatible {fieldCount, plural, =1 {field} other {fields}}, {fieldsInSameFamily} {fieldsInSameFamily, plural, =1 {field} other {fields}} with {fieldsInSameFamily, plural, =1 {a mapping} other {mappings}} in the same family',
+      values: { fieldCount },
+      defaultMessage: '{fieldCount} incompatible {fieldCount, plural, =1 {field} other {fields}}',
     }
   );
 
@@ -391,6 +404,21 @@ export const OTHER_APP_CAPABILITIES_WORK_PROPERLY = i18n.translate(
   }
 );
 
+export const SAME_FAMILY_EMPTY = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyEmptyContent',
+  {
+    defaultMessage:
+      'All of the field mappings and document values in this index are compliant with the Elastic Common Schema (ECS).',
+  }
+);
+
+export const SAME_FAMILY_EMPTY_TITLE = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyEmptyTitle',
+  {
+    defaultMessage: 'All field mappings and values are ECS compliant',
+  }
+);
+
 export const PAGES_DISPLAY_EVENTS = i18n.translate(
   'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.pagesDisplayEventsMessage',
   {
@@ -428,6 +456,13 @@ export const ECS_IS_A_PERMISSIVE_SCHEMA = i18n.translate(
   }
 );
 
+export const SAME_FAMILY = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sameFamilyTab',
+  {
+    defaultMessage: 'Same family',
+  }
+);
+
 export const SOMETIMES_INDICES_CREATED_BY_OLDER = i18n.translate(
   'securitySolutionPackages.ecsDataQualityDashboard.indexProperties.sometimesIndicesCreatedByOlderDescription',
   {

diff --git a/...n/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/pattern/helpers.test.ts b/...n/ecs_data_quality_dashboard/impl/data_quality/data_quality_panel/pattern/helpers.test.ts
@@ -273,6 +273,7 @@ describe('helpers', () => {
           '\n#### Incompatible field mappings - auditbeat-custom-index-1\n\n\n| Field | ECS mapping type (expected) | Index mapping type (actual) | \n|-------|-----------------------------|-----------------------------|\n| host.name | `keyword` | `text`  |\n| source.ip | `ip` | `text`  |\n\n#### Incompatible field values - auditbeat-custom-index-1\n\n\n| Field | ECS values (expected) | Document values (actual) | \n|-------|-----------------------|--------------------------|\n| event.category | `authentication`, `configuration`, `database`, `driver`, `email`, `file`, `host`, `iam`, `intrusion_detection`, `malware`, `network`, `package`, `process`, `registry`, `session`, `threat`, `vulnerability`, `web` | `an_invalid_category` (2),\n`theory` (1) |\n\n',
         ],
         pattern: 'auditbeat-*',
+        sameFamily: 0,
       },
     };
     const isILMAvailable = true;

diff --git a/...cs_data_quality_dashboard/impl/data_quality/data_quality_panel/stat_label/translations.ts b/...cs_data_quality_dashboard/impl/data_quality/data_quality_panel/stat_label/translations.ts
@@ -102,6 +102,23 @@ export const INDICES = i18n.translate(
   }
 );
 
+export const SAME_FAMILY = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.statLabels.sameFamilyLabel',
+  {
+    defaultMessage: 'Same family',
+  }
+);
+
+export const SAME_FAMILY_PATTERN_TOOL_TIP = (pattern: string) =>
+  i18n.translate(
+    'securitySolutionPackages.ecsDataQualityDashboard.statLabels.sameFamilyPatternToolTip',
+    {
+      values: { pattern },
+      defaultMessage:
+        'The total count of fields in the same family as the type specified by ECS, in indices matching the {pattern} pattern',
+    }
+  );
+
 export const SIZE = i18n.translate(
   'securitySolutionPackages.ecsDataQualityDashboard.statLabels.sizeLabel',
   {
@@ -166,6 +183,14 @@ export const TOTAL_INDICES_TOOL_TIP = i18n.translate(
   }
 );
 
+export const TOTAL_SAME_FAMILY_TOOL_TIP = i18n.translate(
+  'securitySolutionPackages.ecsDataQualityDashboard.statLabels.totalSameFamilyToolTip',
+  {
+    defaultMessage:
+      'The total count of fields in the same family as the ECS type, in all indices that were checked',
+  }
+);
+
 export const TOTAL_SIZE_TOOL_TIP = i18n.translate(
   'securitySolutionPackages.ecsDataQualityDashboard.statLabels.totalSizeToolTip',
   {