[7.17] [esArchiver] Automatically cleanup SO indices when SO documents are found in data.json (#159582) #163178
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ound in data.json (elastic#159582) The ultimate goal of this PR is to lay the groundwork to be able to remove the "dynamic" `mappings.json`, which probably should have never existed. With the PR, detecting SO documents in the `data.json` will automatically trigger a cleanup of the SO indices. This, in turn, will allow not having to define "dynamic" saved objects indices (i.e. those with the `$KIBANA_PACKAGE_VERSION` variable in the `mappings.json`). IIUC the idea behind the dynamic indices was to have SO indices that are aligned with the current stack version, avoiding the extra overhead of having to migrate the inserted documents, and reducing overall test times. Nonetheless, what is happening today is: 1. FTR starts ES and Kibana. 2. Kibana creates current version SO indices at startup (empty ones). 3. `esArchiver.load()` processes the `mappings.json`. 3.1. It detects that we are defining SO indices and **deletes** existing saved object indices. 3.2 It then re-creates these indices according to the definitions on `mappings.json`. 4. `esArchiver.load()` processes the `data.json`. Specifically, it inserts SO documents present in `data.json`. 5. `esArchiver.load()` calls the _KibanaMigrator_ to make sure that the inserted documents are up-to-date, hoping they are already aligned with current stack version (which is not always the case, not even with "dynamic" mappings). Two interesting things to note: - Steps 3 to 5 happen whilst Kibana is already started and running. If Kibana queries SO indices during `esArchiver.load()`, and a request to ES is made **right after** 3.2, the result might be elastic#158918. - Having dynamic SO indices' definitions, deleting the "official" indices created by Kibana (3.1), and recreating them hoping to be aligned with current stack version (3.2) is non-sense. We could use the existing SO indices instead, and simply clean them up whenever we are about to insert SO documents. Performing that cleanup is precisely the goal of this PR. Then, in subsequent PRs like https://github.com/elastic/kibana/pull/159397/files, tackling the flaky tests, we'll be able to simply remove the "dynamic" `mappings.json` definitions, causing `esArchiver` to rely on SO indices created by Kibana. Thanks to this PR, the FTR tests won't need to explicitly cleanup saved object indices in the `before` hooks. (cherry picked from commit bbb5fc4) # Conflicts: # packages/kbn-es-archiver/src/lib/indices/create_index_stream.test.mock.ts # packages/kbn-es-archiver/src/lib/indices/create_index_stream.test.ts # packages/kbn-es-archiver/src/lib/indices/create_index_stream.ts # packages/kbn-es-archiver/src/lib/indices/delete_index_stream.ts # packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
dmlemeshko
reviewed
Aug 7, 2023
| await createPromiseFromStreams([ | ||
| createListStream([ | ||
| createStubDocRecord('.kibana_task_manager', 1), | ||
| createStubDocRecord('.kibana_alerting_cases', 2), |
There was a problem hiding this comment.
For myself: we do not have this SO index in 7.17.x? I understand it is just a test and won't affect actual code.
There was a problem hiding this comment.
Good catch, we shouldn't be introducing these 8.8 spoilers in 7.x.
I'll update the test.
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto7.17:Questions ?
Please refer to the Backport tool documentation
\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n\n\nCo-authored-by: Gerard Soldevila "}},{"branch":"8.10","label":"v8.10.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"7.17","label":"v7.17.13","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->