kafka replay speed: remove records-per-fetch configuration #9906
Conversation
dimitarvdimitrov
force-pushed
the
dimitar/ingest/replay-speed/remove-records-per-fetch-config
branch
2 times, most recently
from
763b0e0 to
f697d2b
Compare
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
dimitarvdimitrov
force-pushed
the
dimitar/ingest/replay-speed/remove-records-per-fetch-config
branch
from
f697d2b to
135b872
Compare
Signed-off-by: Dimitar Dimitrov <[email protected]>
Signed-off-by: Dimitar Dimitrov <[email protected]>
| @@ -834,76 +831,6 @@ func TestConcurrentFetchers(t *testing.T) { | |||
| pollFetchesAndAssertNoRecords(t, fetchers) | |||
| }) | |||
|
|
|||
| t.Run("staggered production with one less than multiple of concurrency and records per fetch", func(t *testing.T) { | |||
There was a problem hiding this comment.
now that we don't do exact number of records this test becomes to a large extent non-deterministic. Its purpose was to test an off-by-one condition to the previous test, but achieving this with tests is really hard now because of byte estimations
|
I just noticed that a test hung in CI. Could be worth investigating if we introduced a regression (if so I can't see it just by looking at the code). |
i'm totally beat today. i'll check tomorrow |
Signed-off-by: Dimitar Dimitrov <[email protected]>
oh boy. This test hangs when it fails for some reason. Locally, i see the assertion failing, but the test never terminates. In CI we only observe it never terminating. The test failure was more complicated than I anticipated, but at least now I understand it. I've added comments to explain the asserted values now. |
Signed-off-by: Dimitar Dimitrov <[email protected]>
dimitarvdimitrov
deleted the
dimitar/ingest/replay-speed/remove-records-per-fetch-config
branch
What this PR does
This is the next step after #9892. We want to control only the number of bytes for fetching because that's more indicative of the load we'd receive. It also allows us to fetch in smaller increments when records become big.
Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.