[Merged by Bors] - stabilize e2e tests - take 2

[poszu]

Motivation

Closes #6119

Description

There are a few problems with the e2e tests in CI happening on slow runners, visible in selected logs from a failed run:

[2024-07-10T12:21:02Z INFO  post::prove] generating proof with PoW flags: RandomXFlag(FLAG_HARD_AES | FLAG_JIT | FLAG_ARGON2_SSSE3 | FLAG_ARGON2_AVX2), difficulty (scaled with SU): 864691128455135232, K2PoW difficulty (scaled with SU): 3FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF	{"smesherID": "31d2a7"}
// 7s of RandomX initialization
[2024-07-10T12:21:09Z INFO  post::prove] calculating proof of work for nonces 0..16	{"smesherID": "31d2a7"}
[2024-07-10T12:21:09Z INFO  post::prove] finished k2pow in 0m 0s	{"smesherID": "31d2a7"}
[2024-07-10T12:21:09Z INFO  post::reader] Reading file: /var/folders/ng/t2lj_z6n4h30fn0h2ql_nd0c0000gn/T/Test_CheckpointAfterMerge87768266/003/postdata_0.bin	{"smesherID": "31d2a7"}
[2024-07-10T12:21:09Z INFO  post::prove] found proof for nonce: 2, pow: 0 with [8, 13, 15, 29, 45, 50, 59, 70] indices. It took 0m 0s	{"smesherID": "31d2a7"}
// 2s elapsed because the service is queried infrequently 
[2024-07-10T12:21:11Z INFO  post_service::client] proof is valid (verification took: 0.557015548s)	{"smesherID": "31d2a7"}

leading to the failure:

builder: atx expired: poet round has already started at 2024-07-10 12:21:08 +0000 UTC (now: 2024-07-10 12:21:11.227228 +0000 UTC m=+118.750801938)

PoST proving time

Despite reducing the difficulty of the PoST proof, it can still take significant time to complete. The most time-consuming part now seems to be RandomX VM initialization. In the logs above it took 7s (from 12:21:02 to 12:21:09), in other runs it took 1s - it's very undeterministic and probably depends on how busy the runner is. Unfortunately, this cannot be speeded up, unless we change the k2pow algorithm to something faster in tests (for example via a CLI flag like --testing-mode).

Post Service querying interval

The post-service is queried in 2s intervals. That's too long in tests, where the cycle gap is 3.75s. If the proof were unavailable after 2s, the service would be queried again after the CG finished and the builder would be late for the poet round leading to test failure.

Changes

• configurable post-service query interval. The e2e tests use a 100ms interval.
• TODO: make the CG/epoch longer in e2e tests or think how to speed up the PoST

Test Plan

existing tests pass

TODO

• Explain motivation or link existing issue(s)
• Test changes and document test plan
• Update documentation as needed
• Update changelog as needed

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.7%. Comparing base (f1ff488) to head (3e5ffe2).

Additional details and impacted files

@@           Coverage Diff           @@
##           develop   #6126   +/-   ##
=======================================
  Coverage     81.7%   81.7%           
=======================================
  Files          302     302           
  Lines        32594   32602    +8     
=======================================
+ Hits         26652   26665   +13     
+ Misses        4200    4197    -3     
+ Partials      1742    1740    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[poszu]

bors merge

[spacemesh-bors]

Pull request successfully merged into develop.

Build succeeded:

• ci-status
• systest-status