Multiple failed test runs on unstable branch

[nflaig]

Describe the bug

E2E are unstable and fail randomly due to different reasons

https://github.com/ChainSafe/lodestar/actions/runs/6544810359/job/17772581931#step:6:4013

@lodestar/beacon-node: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
@lodestar/beacon-node:  FAIL  test/e2e/network/reqrespEncode.test.ts > reqresp encoder > assert correct handler switch between metadata v2 and v1
@lodestar/beacon-node:  Test Files  1 failed | 15 passed | 7 skipped (23)
@lodestar/beacon-node: CodeError: Transport (@libp2p/tcp) could not listen on any available address
@lodestar/beacon-node:       Tests  1 failed | 93 passed | 84 skipped (178)
@lodestar/beacon-node:    Start at  09:09:32

https://github.com/ChainSafe/lodestar/actions/runs/6517197158/job/17701469723#step:6:4037

@lodestar/beacon-node: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
@lodestar/beacon-node:  FAIL  test/e2e/api/impl/lightclient/endpoint.test.ts > lightclient api > getOptimisticUpdate()
@lodestar/beacon-node:  Test Files  1 failed | 15 passed | 7 skipped (23)
@lodestar/beacon-node: AssertionError: expected 3 to be 4 // Object.is equality

https://github.com/ChainSafe/lodestar/actions/runs/6517197158/job/17701499293#step:6:1483

@lodestar/beacon-node: Segmentation fault (core dumped)
@lodestar/beacon-node: @lodestar/beacon-node: error Command failed with exit code 139.

Expected behavior

No failures, stable tests

Steps to reproduce

Run E2E on unstable branch

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

unstable (c06f4e5)

[nflaig]

Tests are really flaky right now

• unit tests fail almost 50% of the time with @lodestar/beacon-node: Segmentation fault (core dumped) (vitest related?)
• spec tests consistently run into timeouts (maybe related to running them in parallel)
• sim tests fail due to timeout (this was always an issue)
• e2e sometimes fail due to invalid assertions (seems to happen since we switched to vitest)

[wemeetagain]

Maybe we should consider setting up a server to run a self-hosted runner?
That should get more consistent test results and have a better debugging story, eg for collecting core dumps, logs, etc.

[nflaig]

Maybe we should consider setting up a server to run a self-hosted runner?

That could definitely improve the situation as it is quite noticeable that the CI get less stable if there are multiple concurrent runs

[Aminechakr]

If we can get some metrics about the runner jobs, on how much does it take in terms of cpu/mem during the multiple executions(maybe disk i/o as well if there's any disk writes), it will certainly help on accurately sizing the self-hosted runner.

[nflaig]

Closed in favor of #6358