👨‍🌾 Add a potential temporary fix for flaky tests on macOS #807

azeey · 2021-05-05T23:53:39Z

🦟 Bug fix

Relates to #806

Summary

A potential solution to the problem of flaky tests on macOS is to include all known components in libignition-gazebo.so. I have tested this locally with Address Sanitizer and it seems to work.

Checklist

Signed all commits for DCO
Added tests
Updated documentation (as needed)
Updated migration guide (as needed)
codecheck passed (See contributing)
All tests passed (See test coverage)
While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge

chapulina · 2021-06-30T00:39:48Z

include all known components in libignition-gazebo.so

Do you think we could get a similar effect by including them into the test executable? That would solve the test's issue while leaving the core library intact.

I can't see the results from CI, and it looks like 3 tests failed on Homebrew. I'll sync this with ign-gazebo3 to retrigger CI.

azeey · 2021-06-30T01:09:15Z

I think some, if not all, of the remaining failures might be fixed by gazebosim/gz-common#227.

chapulina · 2021-06-30T01:18:13Z

So you think they're not related to how the components are registered, as explained on #806?

One thing all these test failures have in common is a segfault at startup, even before any console messages are printed.

codecov · 2021-06-30T01:33:07Z

Codecov Report

Merging #807 (6cb1354) into ign-gazebo5 (b45f60f) will decrease coverage by 0.09%.
The diff coverage is 100.00%.

@@               Coverage Diff               @@
##           ign-gazebo5     #807      +/-   ##
===============================================
- Coverage        67.04%   66.95%   -0.10%     
===============================================
  Files              276      276              
  Lines            21049    21023      -26     
===============================================
- Hits             14112    14075      -37     
- Misses            6937     6948      +11

Impacted Files	Coverage Δ
src/SdfEntityCreator.cc	`85.41% <ø> (ø)`
src/SimulationRunner.cc	`91.88% <100.00%> (-2.05%)`	⬇️
src/Barrier.cc	`96.15% <0.00%> (-3.85%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b45f60f...6cb1354. Read the comment docs.

chapulina · 2021-06-30T01:41:35Z

FWIW, there were no homebrew failures in the latest run 😬

azeey · 2021-07-09T19:33:54Z

I've disabled multi-threading in 6ac5110. Let's what this does.

chapulina · 2021-07-09T20:27:10Z

I've disabled multi-threading in 6ac5110. Let's what this does.

It's tough to tell from just one CI run because these failures are very flaky. It would be nice to be able to set something like colcon's --retest-until-fail on CI

scpeters · 2021-07-11T05:20:01Z

I've disabled multi-threading in 6ac5110. Let's what this does.

It's tough to tell from just one CI run because these failures are very flaky. It would be nice to be able to set something like colcon's --retest-until-fail on CI

we currently have a script for running ctest with --rerun-failed if the RERUN_FAILED_TESTS variable is set to 1, though we only use it for gazebo CI

we could add something similar for --retest-until-fail

azeey · 2021-07-12T15:28:03Z

I've disabled multi-threading in 6ac5110. Let's what this does.

It's tough to tell from just one CI run because these failures are very flaky. It would be nice to be able to set something like colcon's --retest-until-fail on CI

we currently have a script for running ctest with --rerun-failed if the RERUN_FAILED_TESTS variable is set to 1, though we only use it for gazebo CI

https://github.com/ignition-tooling/release-tools/blob/master/jenkins-scripts/lib/make_test_rerun_failed.bash#L17

https://github.com/ignition-tooling/release-tools/blob/c819f8682070ee88a9ac80b0983205c5cda5141f/jenkins-scripts/docker/lib/gazebo-base-default.bash#L194

https://github.com/ignition-tooling/release-tools/blob/c819f8682070ee88a9ac80b0983205c5cda5141f/jenkins-scripts/gazebo-default-devel-homebrew-amd64.bash#L13

we could add something similar for --retest-until-fail

Yeah, ctest has repeat-until-fail. How about doing that on a custom RTOOLS_BRANCH? Is gazebo-tooling/release-tools#242 still an issue?

chapulina · 2021-07-12T17:46:58Z

How about doing that on a custom RTOOLS_BRANCH?

+1

Is gazebo-tooling/release-tools#242 still an issue?

I believe it is, but it's flaky and I haven't seen it in a while.

azeey · 2021-07-12T19:50:41Z

Buiding with --repeat-until-fail=100

azeey · 2021-07-13T15:23:02Z

https://build.osrfoundation.org/job/ignition_gazebo-ci-pr_any-homebrew-amd64/5997/ ran each test until it failed with a maximum of 100 runs each. The logs show:

The following tests FAILED:
         39 - UNIT_PeerTracker_TEST (Failed)
        107 - INTEGRATION_network_handshake (Subprocess aborted)
        119 - INTEGRATION_scene_broadcaster_system (SEGFAULT)

I suspect the failure in UNIT_PeerTracker_TEST might be a timing/networking issue (#452). The INTEGRATION_network_handshake test uses multiple threads so it might be a race condition on console stream objects (gazebosim/gz-common#227). I don't have any guesses as to what's causing the segfault in INTEGRATION_scene_broadcaster_system

For comparision, I'll run ign-gazebo3 with --repeat-until-fail and report back.

azeey · 2021-07-13T17:43:20Z

In the interest of time, I only did --repeat-until-fail=10 for the ign-gazebo3 branch: https://build.osrfoundation.org/view/ign-citadel/job/ignition_gazebo-ci-pr_any-homebrew-amd64/6020

The logs show:

The following tests FAILED:
	 39 - UNIT_PeerTracker_TEST (Failed)
	 59 - INTEGRATION_breadcrumbs (SEGFAULT)
	 71 - INTEGRATION_each_new_removed (Subprocess aborted)
	 93 - INTEGRATION_level_manager_runtime_performers (SEGFAULT)
	111 - INTEGRATION_physics_system (SEGFAULT)
	115 - INTEGRATION_pose_publisher_system (SEGFAULT)
	119 - INTEGRATION_scene_broadcaster_system (SEGFAULT)

I'm guessing INTEGRATION_network_handshake would have failed to if I had run it 100 times.

chapulina · 2021-07-13T18:48:12Z

So we're testing 2 different things at once here. The header includes and the worker threads. I think we should examine them separately.

If it turns out the worker threads are the problem, I believe the next step could be to disable multi-threading just for macOS until we have time to try and address the underlying issue. I'm assuming no one has time to debug these failures on a mac right now. I think it's worth sacrificing performance for reliability for now (for Mac users and CI).

azeey · 2022-02-25T17:13:40Z

Now that ign-common 4.5 has been released, we've have had much better homebrew CI https://build.osrfoundation.org/job/ignition_gazebo-ci-ign-gazebo5-homebrew-amd64/ (starting from build 53).

Last 20 builds

INTEGRATION_scene_broadcaster_system segfaulted in build 55 and UNIT_PeerTracker_TEST failed in build 58. I think this is very encouraging. I'll retarget this to ign-gazebo5 for now and run tests to see if the combination of this PR and gazebosim/gz-common#227 show an improvement.

I have attached the test result csv file for posterity: Test Results.csv

Signed-off-by: Addisu Z. Taddese <[email protected]>

chapulina · 2022-03-25T02:11:24Z

@azeey , Edifice will EOL next week. Do you want to retarget this to ign-gazebo6?

azeey · 2022-03-25T19:27:29Z

@azeey , Edifice will EOL next week. Do you want to retarget this to ign-gazebo6?

Yup. Will do.

azeey · 2022-12-22T22:54:09Z

Closing in favor of #1836

github-actions bot added the 🏰 citadel Ignition Citadel label May 5, 2021

chapulina added the macOS macOS support label May 5, 2021

chapulina mentioned this pull request Jul 7, 2021

ign-gazebo3/main homebrew INTEGRATION test failures (segfaults) #630

Closed

azeey added 2 commits February 25, 2022 11:28

Add a potential temporary fix for flaky tests on macOS

7061c55

Signed-off-by: Addisu Z. Taddese <[email protected]>

Disable multi-threading

6cb1354

Signed-off-by: Addisu Z. Taddese <[email protected]>

azeey force-pushed the fix_flaky_macos branch from 75172ca to 6cb1354 Compare February 25, 2022 17:31

azeey changed the base branch from ign-gazebo3 to ign-gazebo5 February 25, 2022 17:31

chapulina added 🏢 edifice Ignition Edifice and removed 🏰 citadel Ignition Citadel labels Mar 24, 2022

chapulina mentioned this pull request Jun 20, 2022

Disable MacOS flakies Citadel #1545

Merged

8 tasks

chapulina added the bug Something isn't working label Jul 23, 2022

azeey mentioned this pull request Sep 22, 2022

Fix macOS test segfaults #1728

Merged

8 tasks

azeey closed this Dec 22, 2022

azeey deleted the fix_flaky_macos branch December 22, 2022 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

👨‍🌾 Add a potential temporary fix for flaky tests on macOS #807

👨‍🌾 Add a potential temporary fix for flaky tests on macOS #807

azeey commented May 5, 2021

chapulina commented Jun 30, 2021

azeey commented Jun 30, 2021

chapulina commented Jun 30, 2021

codecov bot commented Jun 30, 2021 •

edited

Loading

chapulina commented Jun 30, 2021

azeey commented Jul 9, 2021

chapulina commented Jul 9, 2021

scpeters commented Jul 11, 2021

azeey commented Jul 12, 2021

chapulina commented Jul 12, 2021

azeey commented Jul 12, 2021

azeey commented Jul 13, 2021

azeey commented Jul 13, 2021

chapulina commented Jul 13, 2021

azeey commented Feb 25, 2022 •

edited

Loading

chapulina commented Mar 25, 2022

azeey commented Mar 25, 2022

azeey commented Dec 22, 2022

👨‍🌾 Add a potential temporary fix for flaky tests on macOS #807

👨‍🌾 Add a potential temporary fix for flaky tests on macOS #807

Conversation

azeey commented May 5, 2021

🦟 Bug fix

Summary

Checklist

chapulina commented Jun 30, 2021

azeey commented Jun 30, 2021

chapulina commented Jun 30, 2021

codecov bot commented Jun 30, 2021 • edited Loading

Codecov Report

chapulina commented Jun 30, 2021

azeey commented Jul 9, 2021

chapulina commented Jul 9, 2021

scpeters commented Jul 11, 2021

azeey commented Jul 12, 2021

chapulina commented Jul 12, 2021

azeey commented Jul 12, 2021

azeey commented Jul 13, 2021

azeey commented Jul 13, 2021

chapulina commented Jul 13, 2021

azeey commented Feb 25, 2022 • edited Loading

chapulina commented Mar 25, 2022

azeey commented Mar 25, 2022

azeey commented Dec 22, 2022

codecov bot commented Jun 30, 2021 •

edited

Loading

azeey commented Feb 25, 2022 •

edited

Loading