KillRuntime => startAndKillRuntimeRetainsTheState e2e test seems to be flaky

[tillrohrmann]

We have multiple instances where the KillRuntime => startAndKillRuntimeRetainsTheState e2e test fails because after the runtime restarts the subsequent Counter.add call does not complete. There is a suspicious comment https://github.com/restatedev/sdk-test-suite/blob/fb737ce392eaea6038c9103930d95c721bf609e9/src/main/kotlin/dev/restate/sdktesting/tests/KillRuntime.kt#L50-L51 about docker messing up the exposed ports.

Failing test runs:

https://github.com/restatedev/restate/actions/runs/12926599283/job/36050876168?pr=2508
https://github.com/restatedev/restate/actions/runs/13021279660/job/36323280404#step:11:175

[tillrohrmann]

Another instance of the problem https://github.com/restatedev/restate/actions/runs/13072118905/job/36476824341?pr=2600#step:11:231.

[tillrohrmann]

@slinkydeveloper do you have time looking into what's going wrong here?

[tillrohrmann]

Another instance: https://github.com/restatedev/restate/actions/runs/13073056467/job/36479292880?pr=2600#step:11:256

[slinkydeveloper]

I'm gonna push a patch, hoping it fixes the issue. If not, I propose to just go and remove this test, it's not adding so much at the end of the day, and I rather prefer that a test killing the runtime is moved in the runtime tests.

I'm honestly short on ideas here :(

[tillrohrmann]

Could we add a request timeout (something like 5s) and then retry the add invocation?

[slinkydeveloper]

let me try to do that

[slinkydeveloper]

updated the test suite, let's see if it still fails.