[heartbeat][libbeat] Fix potential race in runner lists

[andrewvc]

Fixes #28518 most likely, it's one I haven't personally been able to repro. This makes the runner list synchronous, thus preventing rapid adds/removes from creating simultaneously live plugins of the same type.

There's definitely ways to do this more concurrently, but that sort of optimization/complexity isn't warranted here. Heartbeat can tear down monitors near instantaneously.

Should be tested by setting up, starting, and stopping elastic agent

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-11-09T20:43:33.032+0000

• Duration: 124 min 47 sec

• Commit: 51e68f6

Test stats 🧪

Test	Results
Failed	0
Passed	54199
Skipped	5345
Total	59544

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[andrewvc]

@elasticmachine merge upstream

[andrewvc]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[shahzad31]

we should create a small util for this, this is such a common use case .

[andrewvc]

There's not really a great way to do this since go doesn't support generics (yet). If you have a proposal lmk though

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b sync-rl upstream/sync-rl
git merge upstream/master
git push upstream sync-rl

[jsoriano]

/package

[andrewvc]

Closing for now, I think we've fixed the problems around ID conflicts in other PRs, and I'm a little worried about the more synchronous approach this takes in terms of creating new bugs.