Flaky tests

[nazarhussain]

Describe the bug

There are some flaky tests needed be fixed.

• test/unit/utils/clock.test.ts > util / Clock > getCurrentSlot > 'should return next slot after 11.5s'
• test/e2e/network/gossipsub.test.ts > gossipsub / worker > Publish and receive a blsToExecutionChange
• test/e2e/api/impl/lightclient/endpoint.test.ts > lightclient api > getOptimisticUpdate()
• test/e2e/network/reqresp.test.ts > network / reqresp / worker > Sleep infinite on first byte
• test/e2e/network/reqresp.test.ts > network / reqresp / worker > Sleep infinite on second response chunk
• test/e2e/sync/unknownBlockSync.test.ts > sync / unknown block sync > should do an unknown block parent sync from another BN
• test/e2e/network/gossipsub.test.ts > gossipsub / worker > Publish and receive a voluntaryExit
• test/e2e/network/gossipsub.test.ts > gossipsub / worker > Publish and receive a LightClientFinalityUpdate
• test/e2e/sync/unknownBlockSync.test.ts > sync / unknown block sync > should do an unknown block parent sync from another BN
• test/e2e/network/reqresp.test.ts > network / reqresp / worker > trigger a RESP_TIMEOUT error
• test/e2e/network/reqresp.test.ts > network / reqresp / worker > should send/receive a light client update message
• test/unit/beacon/genericServerTest/config.test.ts > beacon / config > Run generic server test
• Few E2E test failing with Error: Too many messages for missing streams) not instanceof LodestarError: Error: Too many messages for missing streams Related to feat!: split identify and identify-push libp2p/js-libp2p#2387
• Error: Couldn't find a matching chrome browser for tag "123.0.6290.0" on platform "linux"

Expected behavior

All tests should behave as stable they could.

Steps to reproduce

• Run all tests

Additional context

During CI runs on different PRs we found few

Operating system

Linux

Lodestar version or commit hash

unstable

[twoeths]

A lot of failed tests happen with useWorker=true, I think it has something to do with the fact that vitest also run in worker thread
for gossipsub, I suggest to only run on main thread mode for CI in #6368
we could do the same thing in req/resp tests too

I used to be able to stabalize e2e tests in "n-historical" state branch by only use useWorker=false

[nazarhussain]

A lot of failed tests happen with useWorker=true, I think it has something to do with the fact that vitest also run in worker thread

For this reason we already run e2e tests with forks not threads.

lodestar/vitest.base.e2e.config.ts

Line 11 in d6a7a39

pool: "forks",

[wemeetagain]

Sometimes the sim tests still fail on unrelated changes:

• https://github.com/ChainSafe/lodestar/actions/runs/7688253188/job/20949254918?pr=6363
• https://github.com/ChainSafe/lodestar/actions/runs/7694591367/job/20980159851?pr=6369
• https://github.com/ChainSafe/lodestar/actions/runs/7690018577/job/20953270893?pr=6364

[jeluard]

Not directly a sim test error, but somewhat related: Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

[nflaig]

Not directly a sim test error, but somewhat related: Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

This happens since we merged #6410, the CI failed on that PR as well...

[nflaig]

Sim merge tests keep getting stuck and time out after 6h since we merged #6344 (as I noted already in the PR), see recent runs on unstable branch (1, 2).

[nflaig]

The browser tests keep failing due to different reasons, this one looks like a race condition (failed run)? The file has a test suite defined and it passes most of the time.

@lodestar/utils: ⎯⎯⎯⎯⎯⎯ Failed Suites 1 ⎯⎯⎯⎯⎯⎯⎯
@lodestar/utils: FAIL test/unit/err.test.ts [ test/unit/err.test.ts ]
@lodestar/utils: Error: No test suite found in file /home/runner/actions-runner/_work/lodestar/lodestar/packages/utils/test/unit/err.test.ts
@lodestar/utils: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

I would suggest we disable browser tests for now as they don't provide any value like this, and are just annoying to deal with

[nflaig]

test/unit/utils/clock.test.ts > util / Clock > getCurrentSlot > 'should return next slot after 11.5s'

Interestingly I have not seen that one in a while, maybe related to updating vitest? Vitest seems to have some internal issues with timings in general and fails to execute tests deterministically..hope they can improve this in the future

[wemeetagain]

@nazarhussain is this still an issue?

[nazarhussain]

@wemeetagain Not all, but I have seen e2e: lightclient api > getOptimisticUpdate() to be failing sometime.

Will need to review each case explicitly in past runs and check these out.