Why mark maxPendingPersists as deprecated

[QiuMM]

In #4815, the maxPendingPersists in KafkaTuningConfig has been marked as deprecated and the value is always zero so that I can't change it ( I use the version of 0.12.2), but in the up-to-date code, the SeekableStreamIndexTaskTuningConfig#maxPendingPersists can be set again by users. I wonder the internal reason of these changes, any problem of maxPendingPersists? I think this parameter is useful if there are some pending persists, tweak this parameter can help avoid the blocking of ingestion. @pjain1 @jsun98 .

[gianm]

I'm not sure if there was a correctness reason for doing this or not so I can't speak to that.

Its behavior can definitely be unintuitive with regard to memory use though: each pending persist adds another potentially-full IncrementalIndex and so increases memory requirements beyond what you might expect based on a particular maxRowsInMemory setting. But I'm not sure if that's why it was marked deprecated or not. I don't see much discussion about it.

[QiuMM]

@gianm thanks for your reply, tweak the maxRowsInMemory may not help since increase it would also increase the persist time (because more data need to persist), so if during the persist time, more than one potentially-full IncrementalIndex is produced, then there may still be pending persists. In such situation, maxPendingPersists is more useful.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.