load test auto-generated server/clients as part of CI

[BigLep]

Done Criteria

We have a CI step that asserts the "performance" and reliability of edelweiss-generated clients and servers on every PR.

Why Important

Consumers of this library need to have confidence that the generated code meets and maintains a certain performance standard.

For example, it's not fair to ask Hydras (e.g., libp2p/hydra-booster#162) or storetheindex (e.g., ipni/storetheindex#251) to use clients/servers generated by this library until this is done.

Notes

We should be able to show that we can sustain a certain amount of RPS without memory leaks for X minutes and consistent latency within some range.

The exact thresholds and how to pull this off need to be designed/determined.

@BigLep is making things up here, but things I'd want to show:

• Memory: before running this test, memory usage was at X. After Y minutes (or Z million requests), memory usage is still at X (i.e., no memory leaks).
• Latency consistency: After X minutes of running the test (or Y million requests), the latency p0 was less than Z1, p50 was less than Z2, p100 was less than Z3, standard deviation was less than Z4, etc.
  Basically, operating edelweiss-generated code under load should have no memory leaks and not add significant variability to request latency.

[BigLep]

@petar : FYI I further update the description to be more clear about what I think is needed here based on our 2022-03-17 verbal conversation.

[petar]

@BigLep I've added high-load tests for memory leaks and goroutine leaks (this is the first point in the issue above). The tests are part of ipfs/go-delegated-routing#11.

I have also added latency regression checks (this is the second point in the issue). However, as I explained to you verbally, these tests are problematic due to varying conditions in the CI runtime and hardware and if we are to do them correctly, we need to first design and agree on a robust method/workflow for doing speed regression tests. I don't believe PL has such infrastructure in any of its projects, so maybe this is not the place to start. Let's double-check with @aschmahmann

The main difficulty is that speed tests are highly affected by the test runtime environment. Therefore they need dedicated VMs which cannot be preempted and do not share hardware with others. @aschmahmann have we addressed this issue in other projects (in an automated manner)?

One can circumvent this difficulty by checking only for substantial regressions. This approach precludes tracking fine-level statistics like p1, p10, p100, etc. On a related note, @aschmahmann do we have an established library for tracking p-statistics?

[BigLep]

@petar : thanks for the note. Understand about the challenge of timing metrics. I'd rather start somewhere here like you have. Flakiness is the key risk as you called out.

I think some of your questions are likely good topics for PLN Discord in #engineering or #go.

Testground as a Service (when it's ready in Q2) may be part of the answer.

Anyways, glad to hear we have the memory leak and goroutine part solidly in. Thanks!

[petar]

Load testing Delegated Routing is now part of ipfs/go-delegated-routing#11. @BigLep @aschmahmann Should we close this issue?

[BigLep]

@petar : cool. My suggestion would be to link this issue to ipfs/go-delegated-routing#11 and then have this issue be closed when ipfs/go-delegated-routing#11 closes.