[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json #159582
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gsoldevila
added
Team:Core
💚 Build Succeeded
Metrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
|
Pinging @elastic/kibana-core (Team:Core) |
|
Pinging @elastic/kibana-operations (Team:Operations) |
|
Thanks for working on this @gsoldevila! |
rudolf
approved these changes
Jun 15, 2023
jbudz
approved these changes
Jun 15, 2023
|
@gsoldevila thank you for driving the investigation and clear explanation of the current process. Great work!
Great point. We shouldn't make any SO indexes re-creation unless it is a test purpose to keep our tests and Kibana instance closer to end-user actual experience. I might be wrong, but I thought that after .kibana esArchives to kbnArchiver #102552 was completed, we should no longer have a test that I made some runs locally and think that our functional UI tests were not affected with .kibana split becuase they don't modify SO indexes via esArchiver, but |
dmlemeshko
approved these changes
Jun 19, 2023
gsoldevila
added
backport:prev-minor
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 19, 2023
…ound in data.json (elastic#159582) The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic" `mappings.json`, which probably should have never existed. With the PR, detecting SO documents in the `data.json` will automatically trigger a cleanup of the SO indices. This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the `mappings.json`). IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times. Nonetheless, what is happening today is: 1. FTR starts ES and Kibana. 2. Kibana creates current version SO indices at startup (empty ones). 3. `esArchiver.load()` processes the `mappings.json`. 3.1. It detects that we are defining SO indices and **deletes** existing saved object indices. 3.2 It then re-creates these indices according to the definitions on `mappings.json`. 4. `esArchiver.load()` processes the `data.json`. Specifically, it inserts SO documents present in `data.json`. 5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings). Two interesting things to note: - Steps 3 to 5 happen whilst Kibana is already started and running. If Kibana queries SO indices during `esArchiver.load()`, and a request to ES is made **right after** 3.2, the result might be elastic#158918. - Having dynamic SO indices' definitions, deleting the "official" indices created by Kibana (3.1), and recreating them hoping to be aligned with current stack version (3.2) is non-sense. We could use the existing SO indices instead, and simply clean them up whenever we are about to insert SO documents. Performing that cleanup is precisely the goal of this PR. Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic" `mappings.json` definitions, causing `esArchiver` to rely on SO indices created by Kibana. Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the `before` hooks. (cherry picked from commit bbb5fc4)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Jun 19, 2023
… are found in data.json (#159582) (#159910) # Backport This will backport the following commits from `main` to `8.8`: - [[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)](#159582) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-19T11:08:03Z","message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Team:Operations","technical debt","release_note:skip","backport:prev-minor","v8.9.0","FTR","v8.8.2"],"number":159582,"url":"https://github.com/elastic/kibana/pull/159582","mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},"sourceBranch":"main","suggestedTargetBranches":["8.8"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159582","number":159582,"mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Gerard Soldevila <[email protected]>
dmlemeshko
added a commit
that referenced
this pull request
Jun 23, 2023
…exes (#159957) ## Summary Related to #159582 To better understand the PR, we need to have a look at `esArchiver.load` function and how it handles `mappings.json`: - It processes the file and if it finds saved object index (e.g. `.kibana`), it will delete all SO indexes (.kibana, .kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana start. Then esArchiver will re-create all indexes according to defined mappings. In order to keep our tests relevant to the latest code changes, it is good to avoid re-creating SO indexes and use "official" ones whenever it is possible. This PR removes/cleans `mappings.json` files where it seems reasonable. I also deleted few not used archives: - es_archiver/saved_objects/ui_counters - es_archiver/saved_objects/usage_counters - security_solution_cypress/es_archives/empty_kibana
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 23, 2023
…exes (elastic#159957) ## Summary Related to elastic#159582 To better understand the PR, we need to have a look at `esArchiver.load` function and how it handles `mappings.json`: - It processes the file and if it finds saved object index (e.g. `.kibana`), it will delete all SO indexes (.kibana, .kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana start. Then esArchiver will re-create all indexes according to defined mappings. In order to keep our tests relevant to the latest code changes, it is good to avoid re-creating SO indexes and use "official" ones whenever it is possible. This PR removes/cleans `mappings.json` files where it seems reasonable. I also deleted few not used archives: - es_archiver/saved_objects/ui_counters - es_archiver/saved_objects/usage_counters - security_solution_cypress/es_archives/empty_kibana (cherry picked from commit 88871fa)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Jun 23, 2023
…exes (elastic#159957) ## Summary Related to elastic#159582 To better understand the PR, we need to have a look at `esArchiver.load` function and how it handles `mappings.json`: - It processes the file and if it finds saved object index (e.g. `.kibana`), it will delete all SO indexes (.kibana, .kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana start. Then esArchiver will re-create all indexes according to defined mappings. In order to keep our tests relevant to the latest code changes, it is good to avoid re-creating SO indexes and use "official" ones whenever it is possible. This PR removes/cleans `mappings.json` files where it seems reasonable. I also deleted few not used archives: - es_archiver/saved_objects/ui_counters - es_archiver/saved_objects/usage_counters - security_solution_cypress/es_archives/empty_kibana (cherry picked from commit 88871fa)
kibanamachine
added a commit
that referenced
this pull request
Jun 23, 2023
…SO indexes (#159957) (#160435) # Backport This will backport the following commits from `main` to `8.8`: - [FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)](#159957) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-23T16:04:11Z","message":"FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)\n\n## Summary\r\n\r\nRelated to #159582\r\n\r\nTo better understand the PR, we need to have a look at `esArchiver.load`\r\nfunction and how it handles `mappings.json`:\r\n- It processes the file and if it finds saved object index (e.g.\r\n`.kibana`), it will delete all SO indexes (.kibana,\r\n.kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana\r\nstart. Then esArchiver will re-create all indexes according to defined\r\nmappings.\r\n\r\nIn order to keep our tests relevant to the latest code changes, it is\r\ngood to avoid re-creating SO indexes and use \"official\" ones whenever it\r\nis possible. This PR removes/cleans `mappings.json` files where it seems\r\nreasonable. I also deleted few not used archives:\r\n- es_archiver/saved_objects/ui_counters\r\n- es_archiver/saved_objects/usage_counters\r\n- security_solution_cypress/es_archives/empty_kibana","sha":"88871faf60ec5a62c4aea48911883a6d7f937777","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.9.0","v8.8.2","v8.10.0"],"number":159957,"url":"https://github.com/elastic/kibana/pull/159957","mergeCommit":{"message":"FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)\n\n## Summary\r\n\r\nRelated to #159582\r\n\r\nTo better understand the PR, we need to have a look at `esArchiver.load`\r\nfunction and how it handles `mappings.json`:\r\n- It processes the file and if it finds saved object index (e.g.\r\n`.kibana`), it will delete all SO indexes (.kibana,\r\n.kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana\r\nstart. Then esArchiver will re-create all indexes according to defined\r\nmappings.\r\n\r\nIn order to keep our tests relevant to the latest code changes, it is\r\ngood to avoid re-creating SO indexes and use \"official\" ones whenever it\r\nis possible. This PR removes/cleans `mappings.json` files where it seems\r\nreasonable. I also deleted few not used archives:\r\n- es_archiver/saved_objects/ui_counters\r\n- es_archiver/saved_objects/usage_counters\r\n- security_solution_cypress/es_archives/empty_kibana","sha":"88871faf60ec5a62c4aea48911883a6d7f937777"}},"sourceBranch":"main","suggestedTargetBranches":["8.9","8.8"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159957","number":159957,"mergeCommit":{"message":"FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)\n\n## Summary\r\n\r\nRelated to #159582\r\n\r\nTo better understand the PR, we need to have a look at `esArchiver.load`\r\nfunction and how it handles `mappings.json`:\r\n- It processes the file and if it finds saved object index (e.g.\r\n`.kibana`), it will delete all SO indexes (.kibana,\r\n.kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana\r\nstart. Then esArchiver will re-create all indexes according to defined\r\nmappings.\r\n\r\nIn order to keep our tests relevant to the latest code changes, it is\r\ngood to avoid re-creating SO indexes and use \"official\" ones whenever it\r\nis possible. This PR removes/cleans `mappings.json` files where it seems\r\nreasonable. I also deleted few not used archives:\r\n- es_archiver/saved_objects/ui_counters\r\n- es_archiver/saved_objects/usage_counters\r\n- security_solution_cypress/es_archives/empty_kibana","sha":"88871faf60ec5a62c4aea48911883a6d7f937777"}}]}] BACKPORT--> Co-authored-by: Dzmitry Lemechko <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Jun 23, 2023
…SO indexes (#159957) (#160436) # Backport This will backport the following commits from `main` to `8.9`: - [FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)](#159957) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Dzmitry Lemechko","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-23T16:04:11Z","message":"FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)\n\n## Summary\r\n\r\nRelated to #159582\r\n\r\nTo better understand the PR, we need to have a look at `esArchiver.load`\r\nfunction and how it handles `mappings.json`:\r\n- It processes the file and if it finds saved object index (e.g.\r\n`.kibana`), it will delete all SO indexes (.kibana,\r\n.kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana\r\nstart. Then esArchiver will re-create all indexes according to defined\r\nmappings.\r\n\r\nIn order to keep our tests relevant to the latest code changes, it is\r\ngood to avoid re-creating SO indexes and use \"official\" ones whenever it\r\nis possible. This PR removes/cleans `mappings.json` files where it seems\r\nreasonable. I also deleted few not used archives:\r\n- es_archiver/saved_objects/ui_counters\r\n- es_archiver/saved_objects/usage_counters\r\n- security_solution_cypress/es_archives/empty_kibana","sha":"88871faf60ec5a62c4aea48911883a6d7f937777","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.9.0","v8.8.2","v8.10.0"],"number":159957,"url":"https://github.com/elastic/kibana/pull/159957","mergeCommit":{"message":"FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)\n\n## Summary\r\n\r\nRelated to #159582\r\n\r\nTo better understand the PR, we need to have a look at `esArchiver.load`\r\nfunction and how it handles `mappings.json`:\r\n- It processes the file and if it finds saved object index (e.g.\r\n`.kibana`), it will delete all SO indexes (.kibana,\r\n.kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana\r\nstart. Then esArchiver will re-create all indexes according to defined\r\nmappings.\r\n\r\nIn order to keep our tests relevant to the latest code changes, it is\r\ngood to avoid re-creating SO indexes and use \"official\" ones whenever it\r\nis possible. This PR removes/cleans `mappings.json` files where it seems\r\nreasonable. I also deleted few not used archives:\r\n- es_archiver/saved_objects/ui_counters\r\n- es_archiver/saved_objects/usage_counters\r\n- security_solution_cypress/es_archives/empty_kibana","sha":"88871faf60ec5a62c4aea48911883a6d7f937777"}},"sourceBranch":"main","suggestedTargetBranches":["8.9","8.8"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159957","number":159957,"mergeCommit":{"message":"FTR: delete .kibana mappings from archives to avoid recreating SO indexes (#159957)\n\n## Summary\r\n\r\nRelated to #159582\r\n\r\nTo better understand the PR, we need to have a look at `esArchiver.load`\r\nfunction and how it handles `mappings.json`:\r\n- It processes the file and if it finds saved object index (e.g.\r\n`.kibana`), it will delete all SO indexes (.kibana,\r\n.kibana_task_manager, .kibana_alerting_cases, etc.) created on Kibana\r\nstart. Then esArchiver will re-create all indexes according to defined\r\nmappings.\r\n\r\nIn order to keep our tests relevant to the latest code changes, it is\r\ngood to avoid re-creating SO indexes and use \"official\" ones whenever it\r\nis possible. This PR removes/cleans `mappings.json` files where it seems\r\nreasonable. I also deleted few not used archives:\r\n- es_archiver/saved_objects/ui_counters\r\n- es_archiver/saved_objects/usage_counters\r\n- security_solution_cypress/es_archives/empty_kibana","sha":"88871faf60ec5a62c4aea48911883a6d7f937777"}}]}] BACKPORT--> Co-authored-by: Dzmitry Lemechko <[email protected]>
gsoldevila
added
v7.17.13
backport:prev-major
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
gsoldevila
added a commit
to gsoldevila/kibana
that referenced
this pull request
Aug 4, 2023
…ound in data.json (elastic#159582) The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic" `mappings.json`, which probably should have never existed. With the PR, detecting SO documents in the `data.json` will automatically trigger a cleanup of the SO indices. This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the `mappings.json`). IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times. Nonetheless, what is happening today is: 1. FTR starts ES and Kibana. 2. Kibana creates current version SO indices at startup (empty ones). 3. `esArchiver.load()` processes the `mappings.json`. 3.1. It detects that we are defining SO indices and **deletes** existing saved object indices. 3.2 It then re-creates these indices according to the definitions on `mappings.json`. 4. `esArchiver.load()` processes the `data.json`. Specifically, it inserts SO documents present in `data.json`. 5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings). Two interesting things to note: - Steps 3 to 5 happen whilst Kibana is already started and running. If Kibana queries SO indices during `esArchiver.load()`, and a request to ES is made **right after** 3.2, the result might be elastic#158918. - Having dynamic SO indices' definitions, deleting the "official" indices created by Kibana (3.1), and recreating them hoping to be aligned with current stack version (3.2) is non-sense. We could use the existing SO indices instead, and simply clean them up whenever we are about to insert SO documents. Performing that cleanup is precisely the goal of this PR. Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic" `mappings.json` definitions, causing `esArchiver` to rely on SO indices created by Kibana. Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the `before` hooks. (cherry picked from commit bbb5fc4) # Conflicts: # packages/kbn-es-archiver/src/lib/indices/create_index_stream.test.mock.ts # packages/kbn-es-archiver/src/lib/indices/create_index_stream.test.ts # packages/kbn-es-archiver/src/lib/indices/create_index_stream.ts # packages/kbn-es-archiver/src/lib/indices/delete_index_stream.ts # packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
gsoldevila
added a commit
that referenced
this pull request
Aug 7, 2023
…s are found in data.json (#159582) (#163178) # Backport This will backport the following commits from `main` to `7.17`: - [[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)](#159582) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gerard Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-06-19T11:08:03Z","message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa","branchLabelMapping":{"^v8.9.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","Team:Operations","technical debt","release_note:skip","backport:prev-minor","backport:prev-MAJOR","v8.9.0","FTR","v8.8.2","v8.10.0","v7.17.13"],"number":159582,"url":"https://github.com/elastic/kibana/pull/159582","mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},"sourceBranch":"main","suggestedTargetBranches":["7.17"],"targetPullRequestStates":[{"branch":"main","label":"v8.9.0","labelRegex":"^v8.9.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/159582","number":159582,"mergeCommit":{"message":"[esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582)\n\nThe ultimate goal of this PR is to lay the groundwork to be able to\r\nremove the \"dynamic\" `mappings.json`, which probably should have never\r\nexisted.\r\n\r\nWith the PR, detecting SO documents in the `data.json` will\r\nautomatically trigger a cleanup of the SO indices.\r\nThis, in turn, will allow not having to define \"dynamic\" saved objects\r\nindices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the\r\n`mappings.json`).\r\n\r\nIIUC the idea behind the dynamic indices was to have SO indices that are\r\naligned with the current stack version, avoiding the extra overhead of\r\nhaving to migrate the inserted documents, and reducing overall test\r\ntimes.\r\n\r\nNonetheless, what is happening today is:\r\n1. FTR starts ES and Kibana.\r\n2. Kibana creates current version SO indices at startup (empty ones).\r\n3. `esArchiver.load()` processes the `mappings.json`.\r\n3.1. It detects that we are defining SO indices and **deletes** existing\r\nsaved object indices.\r\n3.2 It then re-creates these indices according to the definitions on\r\n`mappings.json`.\r\n4. `esArchiver.load()` processes the `data.json`. Specifically, it\r\ninserts SO documents present in `data.json`.\r\n5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the\r\ninserted documents are up-to-date, hoping they are already aligned with\r\ncurrent stack version (which is not always the case, not even with\r\n\"dynamic\" mappings).\r\n\r\nTwo interesting things to note:\r\n- Steps 3 to 5 happen whilst Kibana is already started and running. If\r\nKibana queries SO indices during `esArchiver.load()`, and a request to\r\nES is made **right after** 3.2, the result might be\r\nhttps://github.com//issues/158918.\r\n- Having dynamic SO indices' definitions, deleting the \"official\"\r\nindices created by Kibana (3.1), and recreating them hoping to be\r\naligned with current stack version (3.2) is non-sense. We could use the\r\nexisting SO indices instead, and simply clean them up whenever we are\r\nabout to insert SO documents.\r\n\r\nPerforming that cleanup is precisely the goal of this PR.\r\nThen, in subsequent PRs like\r\nhttps://github.com//pull/159397/files, tackling the flaky\r\ntests, we'll be able to simply remove the \"dynamic\" `mappings.json`\r\ndefinitions, causing `esArchiver` to rely on SO indices created by\r\nKibana.\r\n\r\nThanks to this PR, the FTR tests won't need to explicitly cleanup saved\r\nobject indices in the `before` hooks.","sha":"bbb5fc4abe7dd530d8248a09a9638cd3438202aa"}},{"branch":"8.8","label":"v8.8.2","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/159910","number":159910,"state":"MERGED","mergeCommit":{"sha":"d2683cba7bfaff545a90c5a7daccb0b731fa1ba0","message":"[8.8] [esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582) (#159910)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.8`:\n- [[esArchiver] Automatically cleanup SO indices when SO documents are\nfound in data.json\n(#159582)](https://github.com/elastic/kibana/pull/159582)\n\n<!--- Backport version: 8.9.7 -->\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT [{\"author\":{\"name\":\"Gerard\nSoldevila\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2023-06-19T11:08:03Z\",\"message\":\"[esArchiver]\nAutomatically cleanup SO indices when SO documents are found in\ndata.json (#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those with the\n`$KIBANA_PACKAGE_VERSION` variable in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the dynamic indices\nwas to have SO indices that are\\r\\naligned with the current stack\nversion, avoiding the extra overhead of\\r\\nhaving to migrate the\ninserted documents, and reducing overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates current version SO indices at\nstartup (empty ones).\\r\\n3. `esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects that we are defining SO indices and\n**deletes** existing\\r\\nsaved object indices.\\r\\n3.2 It then re-creates\nthese indices according to the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()` processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the _KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are up-to-date, hoping they are already\naligned with\\r\\ncurrent stack version (which is not always the case, not\neven with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started and\nrunning. If\\r\\nKibana queries SO indices during `esArchiver.load()`, and\na request to\\r\\nES is made **right after** 3.2, the result might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n- Having\ndynamic SO indices' definitions, deleting the \\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating them hoping to be\\r\\naligned\nwith current stack version (3.2) is non-sense. We could use\nthe\\r\\nexisting SO indices instead, and simply clean them up whenever we\nare\\r\\nabout to insert SO documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of this PR.\\r\\nThen, in subsequent PRs\nlike\\r\\nhttps://github.com//pull/159397/files, tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the \\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver` to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject indices in the `before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\",\"branchLabelMapping\":{\"^v8.9.0$\":\"main\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"Team:Core\",\"Team:Operations\",\"technical\ndebt\",\"release_note:skip\",\"backport:prev-minor\",\"v8.9.0\",\"FTR\",\"v8.8.2\"],\"number\":159582,\"url\":\"https://github.com/elastic/kibana/pull/159582\",\"mergeCommit\":{\"message\":\"[esArchiver]\nAutomatically cleanup SO indices when SO documents are found in\ndata.json (#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those with the\n`$KIBANA_PACKAGE_VERSION` variable in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the dynamic indices\nwas to have SO indices that are\\r\\naligned with the current stack\nversion, avoiding the extra overhead of\\r\\nhaving to migrate the\ninserted documents, and reducing overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates current version SO indices at\nstartup (empty ones).\\r\\n3. `esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects that we are defining SO indices and\n**deletes** existing\\r\\nsaved object indices.\\r\\n3.2 It then re-creates\nthese indices according to the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()` processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the _KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are up-to-date, hoping they are already\naligned with\\r\\ncurrent stack version (which is not always the case, not\neven with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started and\nrunning. If\\r\\nKibana queries SO indices during `esArchiver.load()`, and\na request to\\r\\nES is made **right after** 3.2, the result might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n- Having\ndynamic SO indices' definitions, deleting the \\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating them hoping to be\\r\\naligned\nwith current stack version (3.2) is non-sense. We could use\nthe\\r\\nexisting SO indices instead, and simply clean them up whenever we\nare\\r\\nabout to insert SO documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of this PR.\\r\\nThen, in subsequent PRs\nlike\\r\\nhttps://github.com//pull/159397/files, tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the \\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver` to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject indices in the `before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[\"8.8\"],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v8.9.0\",\"labelRegex\":\"^v8.9.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/159582\",\"number\":159582,\"mergeCommit\":{\"message\":\"[esArchiver]\nAutomatically cleanup SO indices when SO documents are found in\ndata.json (#159582)\\n\\nThe ultimate goal of this PR is to lay the\ngroundwork to be able to\\r\\nremove the \\\"dynamic\\\" `mappings.json`,\nwhich probably should have never\\r\\nexisted.\\r\\n\\r\\nWith the PR,\ndetecting SO documents in the `data.json` will\\r\\nautomatically trigger\na cleanup of the SO indices.\\r\\nThis, in turn, will allow not having to\ndefine \\\"dynamic\\\" saved objects\\r\\nindices (i.e. those with the\n`$KIBANA_PACKAGE_VERSION` variable in\nthe\\r\\n`mappings.json`).\\r\\n\\r\\nIIUC the idea behind the dynamic indices\nwas to have SO indices that are\\r\\naligned with the current stack\nversion, avoiding the extra overhead of\\r\\nhaving to migrate the\ninserted documents, and reducing overall\ntest\\r\\ntimes.\\r\\n\\r\\nNonetheless, what is happening today is:\\r\\n1. FTR\nstarts ES and Kibana.\\r\\n2. Kibana creates current version SO indices at\nstartup (empty ones).\\r\\n3. `esArchiver.load()` processes the\n`mappings.json`.\\r\\n3.1. It detects that we are defining SO indices and\n**deletes** existing\\r\\nsaved object indices.\\r\\n3.2 It then re-creates\nthese indices according to the definitions on\\r\\n`mappings.json`.\\r\\n4.\n`esArchiver.load()` processes the `data.json`. Specifically,\nit\\r\\ninserts SO documents present in `data.json`.\\r\\n5.\n`esArchiver.load()` calls the _KibanaMigrator_ to make sure that\nthe\\r\\ninserted documents are up-to-date, hoping they are already\naligned with\\r\\ncurrent stack version (which is not always the case, not\neven with\\r\\n\\\"dynamic\\\" mappings).\\r\\n\\r\\nTwo interesting things to\nnote:\\r\\n- Steps 3 to 5 happen whilst Kibana is already started and\nrunning. If\\r\\nKibana queries SO indices during `esArchiver.load()`, and\na request to\\r\\nES is made **right after** 3.2, the result might\nbe\\r\\nhttps://github.com//issues/158918.\\r\\n- Having\ndynamic SO indices' definitions, deleting the \\\"official\\\"\\r\\nindices\ncreated by Kibana (3.1), and recreating them hoping to be\\r\\naligned\nwith current stack version (3.2) is non-sense. We could use\nthe\\r\\nexisting SO indices instead, and simply clean them up whenever we\nare\\r\\nabout to insert SO documents.\\r\\n\\r\\nPerforming that cleanup is\nprecisely the goal of this PR.\\r\\nThen, in subsequent PRs\nlike\\r\\nhttps://github.com//pull/159397/files, tackling\nthe flaky\\r\\ntests, we'll be able to simply remove the \\\"dynamic\\\"\n`mappings.json`\\r\\ndefinitions, causing `esArchiver` to rely on SO\nindices created by\\r\\nKibana.\\r\\n\\r\\nThanks to this PR, the FTR tests\nwon't need to explicitly cleanup saved\\r\\nobject indices in the `before`\nhooks.\",\"sha\":\"bbb5fc4abe7dd530d8248a09a9638cd3438202aa\"}},{\"branch\":\"8.8\",\"label\":\"v8.8.2\",\"labelRegex\":\"^v(\\\\d+).(\\\\d+).\\\\d+$\",\"isSourceBranch\":false,\"state\":\"NOT_CREATED\"}]}]\nBACKPORT-->\n\nCo-authored-by: Gerard Soldevila <[email protected]>"}},{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"7.17","label":"v7.17.13","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic"
mappings.json, which probably should have never existed.With the PR, detecting SO documents in the
data.jsonwill automatically trigger a cleanup of the SO indices.This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the
$KIBANA_PACKAGE_VERSIONvariable in themappings.json).IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times.
Nonetheless, what is happening today is:
esArchiver.load()processes themappings.json.3.1. It detects that we are defining SO indices and deletes existing saved object indices.
3.2 It then re-creates these indices according to the definitions on
mappings.json.esArchiver.load()processes thedata.json. Specifically, it inserts SO documents present indata.json.esArchiver.load()calls the KibanaMigrator to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings).Two interesting things to note:
esArchiver.load(), and a request to ES is made right after 3.2, the result might be esArchiver fails with no_shard_available_action_exception after deleting indexes #158918.Performing that cleanup is precisely the goal of this PR.
Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic"
mappings.jsondefinitions, causingesArchiverto rely on SO indices created by Kibana.Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the
beforehooks.