fix: flaky test ClusterMetadataManifestTests #9728

r1walz · 2023-09-04T11:00:58Z

Description

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here.

Related Issues

Resolves #9688

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-09-04T11:30:09Z

Compatibility status:

Checks if related components are compatible with change 7cd5355

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-04T11:45:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/24330/
CommitID: 7cd5355

codecov · 2023-09-04T11:45:56Z

Codecov Report

Merging #9728 (55fec3b) into main (19a13f0) will increase coverage by 0.05%.
Report is 4 commits behind head on main.
The diff coverage is 76.65%.

@@             Coverage Diff              @@
##               main    #9728      +/-   ##
============================================
+ Coverage     71.14%   71.20%   +0.05%     
- Complexity    57973    58070      +97     
============================================
  Files          4825     4827       +2     
  Lines        273348   273539     +191     
  Branches      39841    39858      +17     
============================================
+ Hits         194473   194767     +294     
+ Misses        62523    62447      -76     
+ Partials      16352    16325      -27

Files Changed	Coverage Δ
...on/admin/indices/shrink/TransportResizeAction.java	`47.72% <0.00%> (-19.99%)`	⬇️
...ofile/aggregation/AggregationProfileBreakdown.java	`0.00% <ø> (ø)`
...rch/search/profile/ContextualProfileBreakdown.java	`75.00% <50.00%> (-25.00%)`	⬇️
...search/profile/query/AbstractQueryProfileTree.java	`76.92% <76.92%> (ø)`
...arch/profile/query/ConcurrentQueryProfileTree.java	`88.88% <88.88%> (ø)`
...profile/query/ConcurrentQueryProfileBreakdown.java	`98.59% <98.51%> (-1.41%)`	⬇️
...in/cluster/remotestore/stats/RemoteStoreStats.java	`96.24% <100.00%> (ø)`
...rg/opensearch/index/remote/RemoteSegmentStats.java	`98.24% <100.00%> (ø)`
...ensearch/search/internal/ContextIndexSearcher.java	`73.83% <100.00%> (+0.30%)`	⬆️
...ch/search/profile/AbstractInternalProfileTree.java	`98.11% <100.00%> (+0.03%)`	⬆️
... and 4 more

... and 457 files with indirect coverage changes

peternied

As I understand this test, the manifest is suppose to be mutated during this step, and if it isn't mutated because the value is set to the same value that already was set the test will fail, seems like there was a 1/8 (if the committed value is selected) + 1/2 (if the value was !currentValue) chance this test would fail without this change.

However, this test doesn't seem like a good candidate for randomness. What do you think about remove the random number generation and make 8 sub test cases around each of these fields with deterministic start and end states.

r1walz · 2023-09-04T16:38:57Z

@peternied : I'm aligned on further dividing the test cases for each attribute. @soosinha, your thoughts?

soosinha · 2023-09-05T02:29:51Z

Dividing the test for each field would give better coverage. I am fine with the suggestion.

github-actions · 2023-09-05T06:25:26Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24427/
CommitID: 37dfa0d
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-05T06:43:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24429/
CommitID: 25e14b2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-05T06:44:11Z

Compatibility status:

Checks if related components are compatible with change 37dfa0d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

github-actions · 2023-09-05T09:10:40Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL:
CommitID: ae5cea3
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-05T09:36:19Z

Compatibility status:

Checks if related components are compatible with change ae5cea3

Incompatible components

Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

dblock · 2023-09-05T15:33:29Z

@r1walz please take a look at the gradle failures

r1walz · 2023-09-05T15:53:49Z

@dblock Jenkins was down earlier. I've retriggered the CI actions. Can you please add skip-changelog tag? Thanks.

github-actions · 2023-09-05T18:17:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/24485/
CommitID: 99b4ca7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

r1walz · 2023-09-05T19:40:29Z

#9776 Flaky test. Other gradle check is failing because of OOM issues.

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. Signed-off-by: Rohit Ashiwal <[email protected]>

github-actions · 2023-09-05T21:00:37Z

Compatibility status:

Checks if related components are compatible with change 55fec3b

Incompatible components

Incompatible components: [https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-09-05T21:11:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/24510/
CommitID: 55fec3b

RS146BIJAY · 2023-09-06T08:16:10Z

@r1walz Is there a plan to merge this PR? Seems it is failing continously.

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. Signed-off-by: Rohit Ashiwal <[email protected]> (cherry picked from commit 6fe9388) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. (cherry picked from commit 6fe9388) Signed-off-by: Rohit Ashiwal <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. Signed-off-by: Rohit Ashiwal <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. Signed-off-by: Rohit Ashiwal <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. Signed-off-by: Rohit Ashiwal <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

peternied reviewed Sep 4, 2023

View reviewed changes

soosinha mentioned this pull request Sep 5, 2023

[Remote state] Integrate remote cluster state in publish/commit flow (#9665) #9751

Merged

r1walz force-pushed the fix-confetti branch from 25e14b2 to ae5cea3 Compare September 5, 2023 09:07

soosinha approved these changes Sep 5, 2023

View reviewed changes

peternied approved these changes Sep 5, 2023

View reviewed changes

r1walz force-pushed the fix-confetti branch from ae5cea3 to 99b4ca7 Compare September 5, 2023 15:52

dblock added the skip-changelog label Sep 5, 2023

fix: flaky test ClusterMetadataManifestTests

55fec3b

We're comparing for unequal manifests but randomBoolean() has a high probability of getting clashed because of a small search-space, making test flaky. Let's limit randomization here. Signed-off-by: Rohit Ashiwal <[email protected]>

r1walz force-pushed the fix-confetti branch from 99b4ca7 to 55fec3b Compare September 5, 2023 20:21

sachinpkale approved these changes Sep 6, 2023

View reviewed changes

sachinpkale merged commit 6fe9388 into opensearch-project:main Sep 6, 2023

sachinpkale added the backport 2.x Backport to 2.x branch label Sep 6, 2023

opensearch-trigger-bot bot mentioned this pull request Sep 6, 2023

[Backport 2.x] fix: flaky test ClusterMetadataManifestTests #9804

Merged

r1walz deleted the fix-confetti branch September 6, 2023 22:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: flaky test ClusterMetadataManifestTests #9728

fix: flaky test ClusterMetadataManifestTests #9728

r1walz commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

github-actions bot commented Sep 4, 2023

codecov bot commented Sep 4, 2023 •

edited

Loading

peternied left a comment

r1walz commented Sep 4, 2023

soosinha commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

dblock commented Sep 5, 2023

r1walz commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

r1walz commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

RS146BIJAY commented Sep 6, 2023 •

edited

Loading

fix: flaky test ClusterMetadataManifestTests #9728

fix: flaky test ClusterMetadataManifestTests #9728

Conversation

r1walz commented Sep 4, 2023

Description

Related Issues

Check List

github-actions bot commented Sep 4, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Sep 4, 2023 • edited Loading

Codecov Report

peternied left a comment

Choose a reason for hiding this comment

r1walz commented Sep 4, 2023

soosinha commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 5, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 5, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 5, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

dblock commented Sep 5, 2023

r1walz commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

Gradle Check (Jenkins) Run Completed with:

r1walz commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 5, 2023

Gradle Check (Jenkins) Run Completed with:

RS146BIJAY commented Sep 6, 2023 • edited Loading

codecov bot commented Sep 4, 2023 •

edited

Loading

RS146BIJAY commented Sep 6, 2023 •

edited

Loading