Fix flaky test_in_flight_lost_after_resumed

[fjetter]

This test has been flaky occasionally because it is quite timing sensitive. Due to allow_other_workers there is a remote chance that the key is actually assigned to the wrong worker (haven't investigated why) such that the key is never fetched/in-flight. This makes the logic more robust

Encountered in #6363

[fjetter]

I opened #6373 for the test failure

[github-actions]

Unit Test Results

       15 files  ±0         15 suites  ±0   7h 26m 0s ⏱️ + 9m 36s
  2 804 tests ±0    2 722 ✔️  - 3    78 💤  - 1  4 ❌ +4 
20 794 runs  ±0  19 867 ✔️  - 5  923 💤 +1  4 ❌ +4 

For more details on these failures, see this check.

Results for commit c247d97. ± Comparison against base commit 4d29246.