Failing test: Jest Integration Tests.src/core/server/integration_tests/saved_objects/migrations/group3 - migration v2 migrates saved objects normally with multiple ES nodes #167676
Assignees
Labels
failed-es-promotion
failed-test
A test failure on a tracked branch, potentially flaky-test
Team:Core
Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Comments
kibanamachine
added
the
failed-test
kibanamachine
added
the
Team:Core
|
Pinging @elastic/kibana-core (Team:Core) |
|
New failure: kibana-elasticsearch-snapshot-verify - main |
|
New failure: kibana-elasticsearch-snapshot-verify - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
New failure: kibana-on-merge - main |
|
Skipped. main: dd65ffa |
|
New failure: kibana-elasticsearch-snapshot-verify - main |
|
New failure: kibana-elasticsearch-snapshot-verify - main |
gsoldevila
added a commit
that referenced
this issue
Oct 17, 2024
## Summary Addresses #167676 Addresses #158318 Addresses #163254 Addresses #163255 #### Fix for `multiple_es_nodes.test.ts` Inspired on #193899 1. Start both nodes of ES 8.17.0 with the affected data-archive on separate terminals: 1. Node 01: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Node 02: `yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 2. After ES is ready (without starting Kibana), reindex the index `.kibana_7.13.0_002` 1. Retrieve the settings from the original index via `curl -L 'http://localhost:9200/.kibana_7.13.0_002' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d ''` 2. Create the target index with those settings: ```shell curl -L -X PUT 'http://localhost:9200/.kibana_7.13.0_003' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "mappings": { "properties": { "bar": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "coreMigrationVersion": { "type": "keyword" }, "foo": { "properties": { "status": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "migrationVersion": { "dynamic": "true", "properties": { "bar": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }, "foo": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } } }, "namespace": { "type": "keyword" }, "namespaces": { "type": "keyword" }, "originId": { "type": "keyword" }, "references": { "type": "nested", "properties": { "id": { "type": "keyword" }, "name": { "type": "keyword" }, "type": { "type": "keyword" } } }, "type": { "type": "keyword" }, "updated_at": { "type": "date" } } }, "settings": { "index": { "hidden": "true", "number_of_shards": "1", "number_of_replicas": "0" } } }' ``` 3. Reindex the content: `curl -L 'http://localhost:9200/_reindex' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "source": { "index": ".kibana_7.13.0_002" }, "dest": { "index": ".kibana_7.13.0_003" } }'` 4. Remove the old index and recreate the aliases ```shell curl -L 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -H 'kbn-xsrf: test' -H 'Authorization: Basic c3lzdGVtX2luZGljZXNfc3VwZXJ1c2VyOmNoYW5nZW1l' -d '{ "actions": [ { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0_001" } }, { "remove_index": {"index": ".kibana_7.13.0_002" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana_7.13.0" } }, { "add": { "index": ".kibana_7.13.0_003", "alias": ".kibana" } } ] }' ``` 3. Stop both ES nodes. 4. Compress both archives ```shell cd .es/node01/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*" cd ../../../ cd .es/node02/8.17.0 rm -rf data/nodes # we need to remove this dir or it fails to start again zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*" cd ../../../ ``` 5. Run the tests to confirm that the issue is fixed: `yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/multiple_es_nodes.test.ts` #### Fix for `incompatible_cluster_routing_allocation.test.ts` Inspired on #193741 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.0.0_v1_migrations_sample_data_saved_objects.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/incompatible_cluster_routing_allocation.test.ts ``` #### Fix for `read_batch_size.test.ts` Inspired on #193899 ```shell # 1. Start ES 8.17.0 with the affected data-archive yarn es snapshot --version=8.17.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip # ... after ES has completely started up, stop it. # 2. Compress the archive cd .es/8.17.0 zip -r ../../src/core/server/integration_tests/saved_objects/migrations/archives/8.4.0_with_sample_data_logs.zip data -x "*/\.*" cd ../../ # 3. Run the tests to confirm that the issue is fixed. yarn test:jest_integration src/core/server/integration_tests/saved_objects/migrations/group3/read_batch_size.test.ts ```
|
Addressed by #196641 |
|
New failure: kibana-elasticsearch-snapshot-verify - main |
|
New failure: kibana-elasticsearch-snapshot-verify - 9.0 |
|
New failure: kibana-elasticsearch-snapshot-verify - main |
mistic
added a commit
that referenced
this issue
Jan 30, 2025
mistic
added a commit
that referenced
this issue
Jan 30, 2025
|
Looks like this test is failing only in main because: |
Yeah, this will need to be fixed ES-side. [UPDATE] copy-pasted the wrong error message (buildkite logs) |
|
ES still blocks the upgrade from 8.18 to 9.0, which we cannot fix Kibana side. |
|
ES issue to ensure upgrading from 8.18 to 9.0 is allowed: elastic/elasticsearch#121693 |
TinaHeiligers
added a commit
that referenced
this issue
Feb 5, 2025
fix #167676 for 9.0 promotion steps taken: 1. Start Node01: `yarn es snapshot --version=8.19.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Start Node02: `yarn es snapshot --version=8.19.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 3. Wait for both nodes to start and be ready 4. Stop both ES nodes 5. change into the directory for node01: `cd .es/node01/8.19.0` 6. Remove (or move) the outdated archive: `rm ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip` 7. zip the updated data and store as the original test filename in the test directory: `zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*"` Repeat these steps for node02: 8. `cd ../../../` 9. `cd .es/node02/8.19.0` 10. `rm ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip` 11. `zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*"` 12. `cd ../../../` 13. `git commit -m "Updated archives" && git push` ### Identify risks Data archives have to be updated for every major.last before a new major is released. Should updating the archives not resolve test failures, there may be a bug elsewhere in the stack. If so, carefully inspect the error logs to determine the cause.
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Feb 5, 2025
fix elastic#167676 for 9.0 promotion steps taken: 1. Start Node01: `yarn es snapshot --version=8.19.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Start Node02: `yarn es snapshot --version=8.19.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 3. Wait for both nodes to start and be ready 4. Stop both ES nodes 5. change into the directory for node01: `cd .es/node01/8.19.0` 6. Remove (or move) the outdated archive: `rm ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip` 7. zip the updated data and store as the original test filename in the test directory: `zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*"` Repeat these steps for node02: 8. `cd ../../../` 9. `cd .es/node02/8.19.0` 10. `rm ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip` 11. `zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*"` 12. `cd ../../../` 13. `git commit -m "Updated archives" && git push` ### Identify risks Data archives have to be updated for every major.last before a new major is released. Should updating the archives not resolve test failures, there may be a bug elsewhere in the stack. If so, carefully inspect the error logs to determine the cause. (cherry picked from commit 1c4d0e9)
kibanamachine
added a commit
that referenced
this issue
Feb 5, 2025
…209774) # Backport This will backport the following commits from `main` to `9.0`: - [Update archives for migrations multiple nodes test (#209667)](#209667) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Christiane (Tina) Heiligers","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-02-05T13:57:52Z","message":"Update archives for migrations multiple nodes test (#209667)\n\nfix #167676 for 9.0 promotion\r\n\r\nsteps taken:\r\n1. Start Node01: \r\n`yarn es snapshot --version=8.19.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path .es/node01`\r\n\r\n2. Start Node02: \r\n`yarn es snapshot --version=8.19.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path .es/node02`\r\n\r\n3. Wait for both nodes to start and be ready\r\n\r\n4. Stop both ES nodes\r\n\r\n5. change into the directory for node01:\r\n`cd .es/node01/8.19.0`\r\n\r\n6. Remove (or move) the outdated archive:\r\n`rm\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip`\r\n\r\n7. zip the updated data and store as the original test filename in the\r\ntest directory:\r\n`zip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata -x \"*/\\.*\"`\r\n\r\nRepeat these steps for node02:\r\n\r\n8. `cd ../../../`\r\n9. `cd .es/node02/8.19.0`\r\n10. `rm\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip`\r\n11. `zip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata -x \"*/\\.*\"`\r\n\r\n12. `cd ../../../`\r\n13. `git commit -m \"Updated archives\" && git push`\r\n\r\n### Identify risks\r\n\r\nData archives have to be updated for every major.last before a new major\r\nis released.\r\nShould updating the archives not resolve test failures, there may be a\r\nbug elsewhere in the stack. If so, carefully inspect the error logs to\r\ndetermine the cause.","sha":"1c4d0e99b45e38182af18872c790d701f5da51e6","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:version","v9.1.0"],"title":"Update archives for migrations multiple nodes test","number":209667,"url":"https://github.com/elastic/kibana/pull/209667","mergeCommit":{"message":"Update archives for migrations multiple nodes test (#209667)\n\nfix #167676 for 9.0 promotion\r\n\r\nsteps taken:\r\n1. Start Node01: \r\n`yarn es snapshot --version=8.19.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path .es/node01`\r\n\r\n2. Start Node02: \r\n`yarn es snapshot --version=8.19.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path .es/node02`\r\n\r\n3. Wait for both nodes to start and be ready\r\n\r\n4. Stop both ES nodes\r\n\r\n5. change into the directory for node01:\r\n`cd .es/node01/8.19.0`\r\n\r\n6. Remove (or move) the outdated archive:\r\n`rm\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip`\r\n\r\n7. zip the updated data and store as the original test filename in the\r\ntest directory:\r\n`zip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata -x \"*/\\.*\"`\r\n\r\nRepeat these steps for node02:\r\n\r\n8. `cd ../../../`\r\n9. `cd .es/node02/8.19.0`\r\n10. `rm\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip`\r\n11. `zip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata -x \"*/\\.*\"`\r\n\r\n12. `cd ../../../`\r\n13. `git commit -m \"Updated archives\" && git push`\r\n\r\n### Identify risks\r\n\r\nData archives have to be updated for every major.last before a new major\r\nis released.\r\nShould updating the archives not resolve test failures, there may be a\r\nbug elsewhere in the stack. If so, carefully inspect the error logs to\r\ndetermine the cause.","sha":"1c4d0e99b45e38182af18872c790d701f5da51e6"}},"sourceBranch":"main","suggestedTargetBranches":["9.0"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/209667","number":209667,"mergeCommit":{"message":"Update archives for migrations multiple nodes test (#209667)\n\nfix #167676 for 9.0 promotion\r\n\r\nsteps taken:\r\n1. Start Node01: \r\n`yarn es snapshot --version=8.19.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\n--base-path .es/node01`\r\n\r\n2. Start Node02: \r\n`yarn es snapshot --version=8.19.0 --data-archive\r\nsrc/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\n--base-path .es/node02`\r\n\r\n3. Wait for both nodes to start and be ready\r\n\r\n4. Stop both ES nodes\r\n\r\n5. change into the directory for node01:\r\n`cd .es/node01/8.19.0`\r\n\r\n6. Remove (or move) the outdated archive:\r\n`rm\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip`\r\n\r\n7. zip the updated data and store as the original test filename in the\r\ntest directory:\r\n`zip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip\r\ndata -x \"*/\\.*\"`\r\n\r\nRepeat these steps for node02:\r\n\r\n8. `cd ../../../`\r\n9. `cd .es/node02/8.19.0`\r\n10. `rm\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip`\r\n11. `zip -r\r\n../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip\r\ndata -x \"*/\\.*\"`\r\n\r\n12. `cd ../../../`\r\n13. `git commit -m \"Updated archives\" && git push`\r\n\r\n### Identify risks\r\n\r\nData archives have to be updated for every major.last before a new major\r\nis released.\r\nShould updating the archives not resolve test failures, there may be a\r\nbug elsewhere in the stack. If so, carefully inspect the error logs to\r\ndetermine the cause.","sha":"1c4d0e99b45e38182af18872c790d701f5da51e6"}}]}] BACKPORT--> Co-authored-by: Christiane (Tina) Heiligers <[email protected]>
drewdaemon
pushed a commit
to drewdaemon/kibana
that referenced
this issue
Feb 6, 2025
fix elastic#167676 for 9.0 promotion steps taken: 1. Start Node01: `yarn es snapshot --version=8.19.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip --base-path .es/node01` 2. Start Node02: `yarn es snapshot --version=8.19.0 --data-archive src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip --base-path .es/node02` 3. Wait for both nodes to start and be ready 4. Stop both ES nodes 5. change into the directory for node01: `cd .es/node01/8.19.0` 6. Remove (or move) the outdated archive: `rm ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip` 7. zip the updated data and store as the original test filename in the test directory: `zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_01.zip data -x "*/\.*"` Repeat these steps for node02: 8. `cd ../../../` 9. `cd .es/node02/8.19.0` 10. `rm ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip` 11. `zip -r ../../../src/core/server/integration_tests/saved_objects/migrations/archives/7.13.0_5k_so_node_02.zip data -x "*/\.*"` 12. `cd ../../../` 13. `git commit -m "Updated archives" && git push` ### Identify risks Data archives have to be updated for every major.last before a new major is released. Should updating the archives not resolve test failures, there may be a bug elsewhere in the stack. If so, carefully inspect the error logs to determine the cause.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A test failed on a tracked branch
First failure: CI Build - main
The text was updated successfully, but these errors were encountered: