Add SO migration testing guidance to testing guide #105959

joshdover · 2021-07-16T14:20:32Z

Summary

This adds a new section on how to test Saved Object migrations for simple cases and more complex cases. Includes examples based on new SO test harness from #106116

elasticmachine · 2021-07-16T14:23:07Z

Pinging @elastic/kibana-core (Team:Core)

dev_docs/tutorials/testing_plugins.mdx

TinaHeiligers

Ne content LGTM, thanks for doing this!

joshdover · 2021-08-09T15:31:11Z

@rudolf any additional feedback on this one?

dev_docs/tutorials/saved_objects.mdx

dev_docs/tutorials/testing_plugins.mdx

kibanamachine · 2021-08-10T14:52:08Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: d30dd2a
Storybooks Preview
Flaky suites:
- xpack-kibana-ciGroup13

Test Failures

Kibana Pipeline / general / X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/text·ts.detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type text "is not" operator will return 0 results if it cannot find what it is excluding

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 2 times on tracked branches: https://github.com/elastic/kibana/issues/107911

[00:00:00]       │
[00:00:00]         └-: detection engine api security and spaces enabled
[00:00:00]           └-> "before all" hook in "detection engine api security and spaces enabled"
[00:00:00]           └-: 
[00:00:00]             └-> "before all" hook in ""
[00:00:00]             └-: Detection exceptions data types and operators
[00:00:00]               └-> "before all" hook in "Detection exceptions data types and operators"
[00:00:00]               └-: 
[00:00:00]                 └-> "before all" hook in ""
[00:00:00]                 └-: Rule exception operators for data type text
[00:00:00]                   └-> "before all" hook in "Rule exception operators for data type text"
[00:00:00]                   └-: "is not" operator
[00:00:00]                     └-> "before all" hook for "will return 0 results if it cannot find what it is excluding"
[00:00:00]                     └-> will return 0 results if it cannot find what it is excluding
[00:00:00]                       └-> "before each" hook: global before each for "will return 0 results if it cannot find what it is excluding"
[00:00:00]                       └-> "before each" hook for "will return 0 results if it cannot find what it is excluding"
[00:00:00]                         │ info [o.e.x.i.a.TransportPutLifecycleAction] [node-01] adding index lifecycle policy [.siem-signals-default-migration-cleanup]
[00:00:00]                         │ info [o.e.x.i.a.TransportPutLifecycleAction] [node-01] adding index lifecycle policy [.siem-signals-default]
[00:00:00]                         │ info [o.e.c.m.MetadataIndexTemplateService] [node-01] adding index template [.siem-signals-default] for index patterns [.siem-signals-default-*]
[00:00:00]                         │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.siem-signals-default-000001] creating index, cause [api], templates [.siem-signals-default], shards [1]/[1]
[00:00:00]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.siem-signals-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.siem-signals-default]
[00:00:00]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.siem-signals-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [.siem-signals-default]
[00:00:00]                         │ info [o.e.x.i.a.TransportPutLifecycleAction] [node-01] adding index lifecycle policy [.lists-default]
[00:00:00]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.siem-signals-default-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.siem-signals-default]
[00:00:00]                         │ info [o.e.x.i.a.TransportPutLifecycleAction] [node-01] adding index lifecycle policy [.items-default]
[00:00:00]                         │ info [o.e.c.m.MetadataIndexTemplateService] [node-01] adding template [.lists-default] for index patterns [.lists-default-*]
[00:00:00]                         │ info [o.e.c.m.MetadataIndexTemplateService] [node-01] adding template [.items-default] for index patterns [.items-default-*]
[00:00:00]                         │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.lists-default-000001] creating index, cause [api], templates [.lists-default], shards [1]/[1]
[00:00:00]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.lists-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.lists-default]
[00:00:00]                         │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.items-default-000001] creating index, cause [api], templates [.items-default], shards [1]/[1]
[00:00:01]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.lists-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [.lists-default]
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text] Loading "mappings.json"
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text] Loading "data.json"
[00:00:01]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.items-default-000001] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [.items-default]
[00:00:01]                         │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [text] creating index, cause [api], templates [], shards [1]/[1]
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text] Created index "text"
[00:00:01]                         │ debg [x-pack/test/functional/es_archives/rule_exceptions/text] "text" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:00:01]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.lists-default-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.lists-default]
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text] Indexed 4 docs into "text"
[00:00:01]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.items-default-000001] from [{"phase":"new","action":"complete","name":"complete"}] to [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] in policy [.items-default]
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text_no_spaces] Loading "mappings.json"
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text_no_spaces] Loading "data.json"
[00:00:01]                         │ info [o.e.x.i.IndexLifecycleTransition] [node-01] moving index [.items-default-000001] from [{"phase":"hot","action":"unfollow","name":"branch-check-unfollow-prerequisites"}] to [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}] in policy [.items-default]
[00:00:01]                         │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [text_no_spaces] creating index, cause [api], templates [], shards [1]/[1]
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text_no_spaces] Created index "text_no_spaces"
[00:00:01]                         │ debg [x-pack/test/functional/es_archives/rule_exceptions/text_no_spaces] "text_no_spaces" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:00:01]                         │ info [x-pack/test/functional/es_archives/rule_exceptions/text_no_spaces] Indexed 4 docs into "text_no_spaces"
[00:00:01]                       │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/fNWqn2BARhK5DKiyNxiQng] update_mapping [_doc]
[00:00:06]                       │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/fNWqn2BARhK5DKiyNxiQng] update_mapping [_doc]
[00:00:07]                       │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/fNWqn2BARhK5DKiyNxiQng] update_mapping [_doc]
[00:00:10]                       │ proc [kibana]   log   [13:31:39.994] [info][eventLog][plugins] event logged: {"@timestamp":"2021-08-10T13:31:39.992Z","event":{"provider":"alerting","action":"execute-start","kind":"alert","category":["siem"],"start":"2021-08-10T13:31:39.992Z"},"kibana":{"saved_objects":[{"rel":"primary","type":"alert","id":"489db390-f9df-11eb-8a44-a357ca4fd571","type_id":"siem.signals"}],"task":{"scheduled":"2021-08-10T13:31:39.705Z","schedule_delay":287000000},"server_uuid":"5b2de169-2785-441b-ae8c-186a1936b17d"},"rule":{"id":"489db390-f9df-11eb-8a44-a357ca4fd571","license":"basic","category":"siem.signals","ruleset":"siem"},"message":"alert execution start: \"489db390-f9df-11eb-8a44-a357ca4fd571\"","ecs":{"version":"1.8.0"}}
[00:00:11]                       └- ✖ fail: detection engine api security and spaces enabled  Detection exceptions data types and operators  Rule exception operators for data type text "is not" operator will return 0 results if it cannot find what it is excluding
[00:00:11]                       │      Error: expected 200 "OK", got 409 "Conflict"
[00:00:11]                       │       at Test._assertStatus (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:268:12)
[00:00:11]                       │       at Test._assertFunction (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:283:11)
[00:00:11]                       │       at Test.assert (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:173:18)
[00:00:11]                       │       at assert (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:131:12)
[00:00:11]                       │       at /dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:128:5
[00:00:11]                       │       at Test.Request.callback (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
[00:00:11]                       │       at /dev/shm/workspace/parallel/20/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
[00:00:11]                       │       at IncomingMessage.<anonymous> (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
[00:00:11]                       │       at endReadableNT (internal/streams/readable.js:1317:12)
[00:00:11]                       │       at processTicksAndRejections (internal/process/task_queues.js:82:21)
[00:00:11]                       │ 
[00:00:11]                       │

Stack Trace

Error: expected 200 "OK", got 409 "Conflict"
    at Test._assertStatus (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:268:12)
    at Test._assertFunction (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:283:11)
    at Test.assert (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:173:18)
    at assert (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:131:12)
    at /dev/shm/workspace/parallel/20/kibana/node_modules/supertest/lib/test.js:128:5
    at Test.Request.callback (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:718:3)
    at /dev/shm/workspace/parallel/20/kibana/node_modules/supertest/node_modules/superagent/lib/node/index.js:906:18
    at IncomingMessage.<anonymous> (/dev/shm/workspace/parallel/20/kibana/node_modules/supertest/node_modules/superagent/lib/node/parsers/json.js:19:7)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21)

Metrics [docs]

✅ unchanged

History

💔 Build #144008 failed e444eee
💚 Build #143653 succeeded 1430463
💚 Build #139456 succeeded 18fbffe5ad50075b6549a2deeb614d6234b68f33
💚 Build #139072 succeeded e93dbc1d8d2b4eb52a1b087c91a306d2c417ab7f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-08-10T14:55:24Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

kibanamachine · 2021-08-12T14:57:07Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

kibanamachine · 2021-08-13T15:07:07Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

kibanamachine · 2021-08-16T15:17:07Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

Co-authored-by: Josh Dover <[email protected]>

joshdover added release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.15.0 v8.0.0 labels Jul 16, 2021

joshdover marked this pull request as ready for review July 16, 2021 14:23

joshdover requested a review from a team as a code owner July 16, 2021 14:23

joshdover commented Jul 19, 2021

View reviewed changes

dev_docs/tutorials/testing_plugins.mdx Outdated Show resolved Hide resolved

joshdover marked this pull request as draft July 19, 2021 13:52

TinaHeiligers approved these changes Jul 26, 2021

View reviewed changes

joshdover mentioned this pull request Jul 28, 2021

Move src/core/TESTING.md to mdx #107039

Merged

joshdover added 2 commits August 9, 2021 14:05

Add migration testing guide

0db449e

Update integration example with new test util

1430463

joshdover force-pushed the docs/testing branch from 18fbffe to 1430463 Compare August 9, 2021 12:06

joshdover changed the title ~~Migrate plugin testing guide + add migration testing section~~ Add SO migration testing guidance to testing guide Aug 9, 2021

joshdover marked this pull request as ready for review August 9, 2021 15:23

rudolf approved these changes Aug 10, 2021

View reviewed changes

dev_docs/tutorials/saved_objects.mdx Outdated Show resolved Hide resolved

dev_docs/tutorials/testing_plugins.mdx Outdated Show resolved Hide resolved

joshdover added 2 commits August 10, 2021 13:12

PR comments

696e99b

Merge remote-tracking branch 'upstream/master' into docs/testing

e444eee

joshdover added the auto-backport Deprecated - use backport:version if exact versions are needed label Aug 10, 2021

joshdover enabled auto-merge (squash) August 10, 2021 11:12

Merge remote-tracking branch 'upstream/master' into docs/testing

d30dd2a

joshdover merged commit 283349a into elastic:master Aug 10, 2021

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 10, 2021

Add SO migration testing guidance to testing guide (elastic#105959)

1aad1f8

kibanamachine mentioned this pull request Aug 10, 2021

[7.x] Add SO migration testing guidance to testing guide (#105959) #108055

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 12, 2021

kibanamachine added a commit that referenced this pull request Aug 16, 2021

Add SO migration testing guidance to testing guide (#105959) (#108055)

c92da6d

Co-authored-by: Josh Dover <[email protected]>

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 16, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SO migration testing guidance to testing guide #105959

Add SO migration testing guidance to testing guide #105959

joshdover commented Jul 16, 2021 •

edited

Loading

elasticmachine commented Jul 16, 2021

TinaHeiligers left a comment

joshdover commented Aug 9, 2021

kibanamachine commented Aug 10, 2021

Standard Out

Stack Trace

kibanamachine commented Aug 10, 2021

kibanamachine commented Aug 12, 2021

kibanamachine commented Aug 13, 2021

kibanamachine commented Aug 16, 2021

Add SO migration testing guidance to testing guide #105959

Add SO migration testing guidance to testing guide #105959

Conversation

joshdover commented Jul 16, 2021 • edited Loading

Summary

elasticmachine commented Jul 16, 2021

TinaHeiligers left a comment

Choose a reason for hiding this comment

joshdover commented Aug 9, 2021

kibanamachine commented Aug 10, 2021

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Metrics [docs]

History

kibanamachine commented Aug 10, 2021

💚 Backport successful

kibanamachine commented Aug 12, 2021

kibanamachine commented Aug 13, 2021

kibanamachine commented Aug 16, 2021

joshdover commented Jul 16, 2021 •

edited

Loading