improve tiered storage system test

[showuon]

Type of change

Select the type of your PR

• Enhancement=
• Documentation

Description

1. In testTieredStorageWithAivenPlugin, we'll wait for the log uploaded or deleted in remote storage, this can be speeded up by reducing remote.log.manager.task.interval.ms config value (default is 30 sec).
2. In testTieredStorageWithAivenPlugin, we'll wait for the local log got deleted, this can be speeded up by reducing log.retention.check.interval.ms config value (default is 5 min).
3. Update the dev guide because we don't have testCustomAndUpdatedValues test in KafkaST now.

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[im-konge]

LGTM, thanks

[see-quick]

LGTM, just out of curiosity how quicker will test go now? (compared to previous runs?) I assumed you have tested it.

[showuon]

LGTM, just out of curiosity how quicker will test go now? (compared to previous runs?) I assumed you have tested it.

@see-quick , I've removed the test terminals. But the most of time in the test is:

1. waiting for data uploaded to S3
2. waiting for local log deletion
3. waiting for S3 data is empty

For (1), the related config is: remote.log.manager.task.interval.ms, whose default is 30 sec. So we might need additional 30 sec if we missed the previous check.
For (2), the topic config (file.delete.delay.ms=1000) has fixed it.
For (3), the related config is log.retention.check.interval.ms, whose default is 5 min. This has the most influence because if we missed the previous check, the test will need to wait for another 5 min.

So, basically, in the worst case, we can save (1) 30 sec + (3)300 sec - (new 1) 5 sec - (new 3) 5 sec = 320 sec, which is more than 5 mins.

[see-quick]

LGTM, just out of curiosity how quicker will test go now? (compared to previous runs?) I assumed you have tested it.

@see-quick , I've removed the test terminals. But the most of time in the test is:

1. waiting for data uploaded to S3
2. waiting for local log deletion
3. waiting for S3 data is empty

For (1), the related config is: remote.log.manager.task.interval.ms, whose default is 30 sec. So we might need additional 30 sec if we missed the previous check. For (2), the topic config (file.delete.delay.ms=1000) has fixed it. For (3), the related config is log.retention.check.interval.ms, whose default is 5 min. This has the most influence because if we missed the previous check, the test will need to wait for another 5 min.

So, basically, in the worst case, we can save (1) 30 sec + (3)300 sec - (new 1) 5 sec - (new 3) 5 sec = 320 sec, which is more than 5 mins.

Thanks, and good job with such optimization! 👍

[see-quick]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[scholzj]

Thanks for the PR.