Skip to content

Commit

Permalink
[8.x] [Security Solution] [Bug] Index Values are not available in dro…
Browse files Browse the repository at this point in the history 
…pdown under New Index Enter for Knowledge Base. (elastic#199610) (elastic#199990) (elastic#200609)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] [Bug] Index Values are not available in dropdown
under New Index Enter for Knowledge Base. (elastic#199610)
(elastic#199990)](elastic#199990)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ievgen
Sorokopud","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-18T16:23:05Z","message":"[Security
Solution] [Bug] Index Values are not available in dropdown under New
Index Enter for Knowledge Base. (elastic#199610) (elastic#199990)\n\n##
Summary\r\n\r\nThis PR fixes the issue described
here\r\nhttps://github.com/elastic/issues/199610#issuecomment-2473393109\r\n\r\nIn
short, we do not show all indices with `semantic_text` fields.
We\r\nonly show those which have `semantic_text` fields named
`content`.\r\n\r\n### Testing notes\r\n\r\nAdd the index with the
`semantic_text` field by running this command in\r\ndev
tools:\r\n\r\n```\r\nPUT test_index\r\n{\r\n \"mappings\": {\r\n
\"properties\": {\r\n \"attachment\": {\r\n \"properties\": {\r\n
\"content\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n
\"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n
}\r\n }\r\n },\r\n \"content_length\": {\r\n \"type\": \"long\"\r\n
},\r\n \"content_type\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n
\"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n
}\r\n }\r\n },\r\n \"format\": {\r\n \"type\": \"text\",\r\n \"fields\":
{\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\":
256\r\n }\r\n }\r\n },\r\n \"language\": {\r\n \"type\": \"text\",\r\n
\"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n
\"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"content2\":
{\r\n \"type\": \"semantic_text\",\r\n \"inference_id\":
\"elastic-security-ai-assistant-elser2\"\r\n }\r\n
}\r\n}\r\n}\r\n```\r\n\r\nYou should see the `test_index` index in the
`Knowledge Base > New >\r\nIndex > Index (dropdown)`.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Steph Milovic
<[email protected]>","sha":"5ab39e5a413f426da232723317786321a7d57fc7","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","Team:
SecuritySolution","Team:Security Generative
AI","backport:version","v8.17.0","v8.16.1"],"title":"[Security Solution]
[Bug] Index Values are not available in dropdown under New Index Enter
for Knowledge Base.
(elastic#199610)","number":199990,"url":"https://github.com/elastic/kibana/pull/199990","mergeCommit":{"message":"[Security
Solution] [Bug] Index Values are not available in dropdown under New
Index Enter for Knowledge Base. (elastic#199610) (elastic#199990)\n\n##
Summary\r\n\r\nThis PR fixes the issue described
here\r\nhttps://github.com/elastic/issues/199610#issuecomment-2473393109\r\n\r\nIn
short, we do not show all indices with `semantic_text` fields.
We\r\nonly show those which have `semantic_text` fields named
`content`.\r\n\r\n### Testing notes\r\n\r\nAdd the index with the
`semantic_text` field by running this command in\r\ndev
tools:\r\n\r\n```\r\nPUT test_index\r\n{\r\n \"mappings\": {\r\n
\"properties\": {\r\n \"attachment\": {\r\n \"properties\": {\r\n
\"content\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n
\"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n
}\r\n }\r\n },\r\n \"content_length\": {\r\n \"type\": \"long\"\r\n
},\r\n \"content_type\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n
\"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n
}\r\n }\r\n },\r\n \"format\": {\r\n \"type\": \"text\",\r\n \"fields\":
{\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\":
256\r\n }\r\n }\r\n },\r\n \"language\": {\r\n \"type\": \"text\",\r\n
\"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n
\"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"content2\":
{\r\n \"type\": \"semantic_text\",\r\n \"inference_id\":
\"elastic-security-ai-assistant-elser2\"\r\n }\r\n
}\r\n}\r\n}\r\n```\r\n\r\nYou should see the `test_index` index in the
`Knowledge Base > New >\r\nIndex > Index (dropdown)`.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Steph Milovic
<[email protected]>","sha":"5ab39e5a413f426da232723317786321a7d57fc7"}},"sourceBranch":"main","suggestedTargetBranches":["8.x","8.16"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199990","number":199990,"mergeCommit":{"message":"[Security
Solution] [Bug] Index Values are not available in dropdown under New
Index Enter for Knowledge Base. (elastic#199610) (elastic#199990)\n\n##
Summary\r\n\r\nThis PR fixes the issue described
here\r\nhttps://github.com/elastic/issues/199610#issuecomment-2473393109\r\n\r\nIn
short, we do not show all indices with `semantic_text` fields.
We\r\nonly show those which have `semantic_text` fields named
`content`.\r\n\r\n### Testing notes\r\n\r\nAdd the index with the
`semantic_text` field by running this command in\r\ndev
tools:\r\n\r\n```\r\nPUT test_index\r\n{\r\n \"mappings\": {\r\n
\"properties\": {\r\n \"attachment\": {\r\n \"properties\": {\r\n
\"content\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n
\"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n
}\r\n }\r\n },\r\n \"content_length\": {\r\n \"type\": \"long\"\r\n
},\r\n \"content_type\": {\r\n \"type\": \"text\",\r\n \"fields\": {\r\n
\"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\": 256\r\n
}\r\n }\r\n },\r\n \"format\": {\r\n \"type\": \"text\",\r\n \"fields\":
{\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n \"ignore_above\":
256\r\n }\r\n }\r\n },\r\n \"language\": {\r\n \"type\": \"text\",\r\n
\"fields\": {\r\n \"keyword\": {\r\n \"type\": \"keyword\",\r\n
\"ignore_above\": 256\r\n }\r\n }\r\n }\r\n }\r\n },\r\n \"content2\":
{\r\n \"type\": \"semantic_text\",\r\n \"inference_id\":
\"elastic-security-ai-assistant-elser2\"\r\n }\r\n
}\r\n}\r\n}\r\n```\r\n\r\nYou should see the `test_index` index in the
`Knowledge Base > New >\r\nIndex > Index (dropdown)`.\r\n\r\n###
Checklist\r\n\r\nDelete any items that are not applicable to this
PR.\r\n\r\n- [ ] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: Steph Milovic
<[email protected]>","sha":"5ab39e5a413f426da232723317786321a7d57fc7"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.16","label":"v8.16.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ievgen Sorokopud <[email protected]>
  • Loading branch information
@kibanamachine @e40pud
kibanamachine and e40pud authored Nov 18, 2024
1 parent cdd8d06 commit 5b40d50
Show file tree
Hide file tree
Showing 2 changed files with 68 additions and 14 deletions.
74 changes: 64 additions & 10 deletions ...plugins/elastic_assistant/server/routes/knowledge_base/get_knowledge_base_indices.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,15 @@ import { getGetKnowledgeBaseIndicesRequest } from '../../__mocks__/request';

const mockFieldCaps = {
indices: [
'.ds-logs-endpoint.alerts-default-2024.10.31-000001',
'.ds-metrics-endpoint.metadata-default-2024.10.31-000001',
'.internal.alerts-security.alerts-default-000001',
'.ds-.items-default-2024.11.12-000001',
'.ds-.lists-default-2024.11.12-000001',
'.ds-logs-endpoint.alerts-default-2024.11.12-000001',
'.ds-logs-endpoint.events.process-default-2024.11.12-000001',
'gtr-1',
'gtr-with-bug',
'gtr-with-semantic-1',
'metrics-endpoint.metadata_current_default',
'semantic-index-1',
'semantic-index-2',
'search-elastic-security-docs',
],
fields: {
content: {
Expand All @@ -27,9 +30,12 @@ const mockFieldCaps = {
searchable: false,
aggregatable: false,
indices: [
'.ds-logs-endpoint.alerts-default-2024.10.31-000001',
'.ds-metrics-endpoint.metadata-default-2024.10.31-000001',
'.internal.alerts-security.alerts-default-000001',
'.ds-.items-default-2024.11.12-000001',
'.ds-.lists-default-2024.11.12-000001',
'.ds-logs-endpoint.alerts-default-2024.11.12-000001',
'.ds-logs-endpoint.events.process-default-2024.11.12-000001',
'gtr-1',
'gtr-with-bug',
'metrics-endpoint.metadata_current_default',
],
},
Expand All @@ -38,7 +44,55 @@ const mockFieldCaps = {
metadata_field: false,
searchable: true,
aggregatable: false,
indices: ['semantic-index-1', 'semantic-index-2'],
indices: ['gtr-with-semantic-1'],
},
},
ai_embeddings: {
unmapped: {
type: 'unmapped',
metadata_field: false,
searchable: false,
aggregatable: false,
indices: [
'.ds-.items-default-2024.11.12-000001',
'.ds-.lists-default-2024.11.12-000001',
'.ds-logs-endpoint.alerts-default-2024.11.12-000001',
'.ds-logs-endpoint.events.process-default-2024.11.12-000001',
'gtr-1',
'gtr-with-semantic-1',
'metrics-endpoint.metadata_current_default',
],
},
semantic_text: {
type: 'semantic_text',
metadata_field: false,
searchable: true,
aggregatable: false,
indices: ['gtr-with-bug', 'search-elastic-security-docs'],
},
},
semantic_text: {
unmapped: {
type: 'unmapped',
metadata_field: false,
searchable: false,
aggregatable: false,
indices: [
'.ds-.items-default-2024.11.12-000001',
'.ds-.lists-default-2024.11.12-000001',
'.ds-logs-endpoint.alerts-default-2024.11.12-000001',
'.ds-logs-endpoint.events.process-default-2024.11.12-000001',
'gtr-1',
'gtr-with-semantic-1',
'metrics-endpoint.metadata_current_default',
],
},
semantic_text: {
type: 'semantic_text',
metadata_field: false,
searchable: true,
aggregatable: false,
indices: ['search-elastic-security-docs'],
},
},
},
Expand Down Expand Up @@ -66,7 +120,7 @@ describe('Get Knowledge Base Status Route', () => {

expect(response.status).toEqual(200);
expect(response.body).toEqual({
indices: ['semantic-index-1', 'semantic-index-2'],
indices: ['gtr-with-bug', 'gtr-with-semantic-1', 'search-elastic-security-docs'],
});
expect(context.core.elasticsearch.client.asCurrentUser.fieldCaps).toBeCalledWith({
index: '*',
Expand Down
8 changes: 4 additions & 4 deletions x-pack/plugins/elastic_assistant/server/routes/knowledge_base/get_knowledge_base_indices.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,10 @@ export const getKnowledgeBaseIndicesRoute = (router: ElasticAssistantPluginRoute
include_unmapped: true,
});

const indices = res.fields.content?.semantic_text?.indices;
if (indices) {
body.indices = Array.isArray(indices) ? indices : [indices];
}
body.indices = Object.values(res.fields)
.flatMap((value) => value.semantic_text?.indices ?? [])
.filter((value, index, self) => self.indexOf(value) === index)
.sort();

return response.ok({ body });
} catch (err) {
Expand Down

0 comments on commit 5b40d50

Please sign in to comment.