[Security Solution] New Data Quality dashboard Same family category

[andrew-goldstein]

[Security Solution] New Data Quality dashboard Same family category

This PR introduces a new Same family category to the Data Quality dashboard.

Fields with the same family tag that were previously counted in the Incompatible fields category are now counted in the new Same family category, per the annotated screenshot below:

Above - Left: previously, fields with the same family tag were counted as Incompatible fields, Right: in this PR, those fields are counted in the new Same family category

The annotations on the Right side of the screenshot above highlight (in this example):

• The total Incompatible fields count was reduced
  • Some patterns, like auditbeat-* have significant reductions
• The Incompatible fields count was reduced by one field
• The new Same family category has one field
• The agent.type field moved from the Incompatible fields category to the Same family category

Details

The new Same family category

Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.

The color of the Same family category is the same as the Custom fields category in badges and charts, per the screenshot below:

Above: The Same family tab's badge and chart legend color is the same as the Custom fields category

The new Same family tab

This PR introduces a new Same family tab, as shown in the screenshot below:

Above: The new Same family tab is selected

In the screenshot above:

• The callout includes a description of fields in the same family (moved from the Incompatible fields tab)
• The constant_keyword text, yellow in previous versions (when it appered in the Incompatible fields tab), is blue
• Only one action, Copy to clipboard is available in the Same family tab. The remaining text in this Details section is the markdown copied to the clipboard for the example above:

auditbeat-custom-index-1

Result	Index	Docs	Incompatible fields	ILM Phase	Size
❌	auditbeat-custom-index-1	2 (0.0%)	3	unmanaged	13.1KB

Incompatible fields 3 Same family 1 Custom fields 4 ECS compliant fields 2 All fields 10

1 Same family field mapping

This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match.

Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.

Same family field mappings - auditbeat-custom-index-1

Field	ECS mapping type (expected)	Index mapping type (actual)
agent.type	keyword	constant_keyword same family

Desk testing

1. Navigate to Dev Tools > Console

2. Execute the queries below:

Queries to create the `auditbeat-custom-index-1` example in this PR description

DELETE auditbeat-custom-index-1

PUT auditbeat-custom-index-1

PUT auditbeat-custom-index-1/_mapping
{
"properties": {
  "@timestamp": {
    "type": "date"
  },
  "agent.type": {
    "type": "constant_keyword"
  },
  "event.category": {
    "type": "constant_keyword"
  }
}
}

POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:41:49.668Z",
"host": {
  "name": "foo"
},
"event": {
  "category": "an_invalid_category"
},
"some.field": "this",
"source": {
  "port": 90210,
  "ip": "10.1.2.3"
}
}

POST auditbeat-custom-index-1/_doc
{
"@timestamp": "2023-02-06T09:42:22.123Z",
"host": {
  "name": "bar"
},
"event": {
  "category": "an_invalid_category"
},
"some.field": "space",
"source": {
  "port": 867,
  "ip": "10.9.8.7"
}
}

3. Navigate to Security > Dashboards > Data Quality

4. Expand the auditbeat-custom-index-1 index

Expected results

• The sum of the category badge counts, 3 + 1 + 4 + 2, equals the total number of fields in the All fields [10] category:
  • Incompatible fields [3]
  • Same family [1]
  • Custom fields [4]
  • ECS compliant fields [2]
• The Incompatible fields callout title is 3 incompatible fields
• The Incompatible fields callout does NOT include a description of same family fields
• The Incompatible fields tab displays two fields with incompatible mappings
  • host.name (keyword vs text)
  • source.ip (ip vs text)
• The Incompatible fields tab displays one field with incompatible field values
  • event.category (an_invalid_category)

5. Click the Summary tab

Expected results

• The Summary tab is focused
• The chart legend includes a Same family entry, with a count of 1
• The Same family tab's badge color is the same as the Custom fields category
• The Same family chart legend color is the same as the Custom fields category

6. Click the Same family legend item

Expected results

• The Same family tab is focused
• One field, agent.type, is displayed
  • The constant_keyword Index mapping type is blue
  • The same family badge is yellow

7. Click Copy to clipboard

Expected result

• The expected markdown is copied to the clipboard:

### auditbeat-custom-index-1

| Result | Index | Docs | Incompatible fields | ILM Phase | Size |
|--------|-------|------|---------------------|-----------|------|
| ❌ | auditbeat-custom-index-1 | 2 (0.0%) | 3 | `unmanaged` | 12.9KB |


### **Incompatible fields** `3` **Same family** `1` **Custom fields** `4` **ECS compliant fields** `2` **All fields** `10`

#### 1 Same family field mapping

This field is defined by the Elastic Common Schema (ECS), version 8.6.1, but its mapping type doesn't exactly match.

Fields with mappings in the same family have exactly the same search behavior as the type specified by ECS, but may have different space usage or performance characteristics.


#### Same family field mappings - auditbeat-custom-index-1


| Field | ECS mapping type (expected) | Index mapping type (actual) |
|-------|-----------------------------|-----------------------------|
| agent.type | `keyword` | `constant_keyword` `same family` |

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[andrew-goldstein]

@elasticmachine merge upstream

[angorayc]

Tested locally, all looks good, LGTM 🚀 Thank you @andrew-goldstein

[michaelolo24]

super super nit: the I is accidentally capitalized I think WIldcard => Wildcard

[michaelolo24]

Investigations codeowner review. Nice work and thanks for the great description as always!

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: e60631a

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	4628	4631	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	13.0MB	13.0MB	+4.3KB

History

• 💚 Build #164104 succeeded 60d5f45092d02bb7a549b57e16891cd169ed005f
• 💛 Build #163334 was flaky 1d4433eef11ccfba66a0897061bbe600ed55a961

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein