Parallel and synchronized test execution for loadscraper.TestScrape

[michalpristas]

Description:
This is not a pretty refactor but I believe this is proper way to fix the test.

I was running into 0 is not lower than 0 error when Processor Queue Length on my system was properly reported as 0.
I added a special case not to fail in this case.

Then the main fix here is making test cases run in parallel as concurrent as possible.
Windows is special case when it comes to reporting CPU metrics as it uses sampling. This sampling is happening every 5 seconds.
While sampling it loads current load from counter and does 1/5/15 minutes average.
As tests are running sequentially, this current load can be different.

To work around this each test case is started in a goroutine while main test goroutine waits for all of them to start.
Each subtest waits for signal to start processing. This signal comes from main goroutine.
This way we achieve as close execution as possible. However there's no guarantee but it should be way more stable than it was.
To make test even more realistic, I changed sequencing frequency to 500 milliseconds in a test and let 3 seconds for scraper to do some actual work and not evaluate on a single scrape.

I ran more than 1200 consecutive executions without failure on my machine.

Link to tracking Issue: #10030

Testing: Unit test

Documentation:

[michalpristas]

hey @dmitryax pinging just not to let this PR die stale. ay feedback would be appreciated

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[dmitryax]

Hi @michalpristas , I'm sorry I missed the PR. It LGTM. Can you please rebase it?

[michalpristas]

hey @dmitryax thanks for getting back. will rebase soon