Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disabled compaction during reconnect leads to a large number of files/descriptors #17065

Closed
OlegMazurov opened this issue Dec 13, 2024 · 1 comment · Fixed by #17129
Closed
Labels
Performance Issues related to performance concerns. Platform Data Structures Platform Reconnect Platform Virtual Map Platform Tickets pertaining to the platform

Comments

@OlegMazurov
Copy link
Contributor

Problem

When new data is received into a VirtualMap during reconnect it is regularly flushed but never compacted if the new status after reconnect is BEHIND. The number of files in the data base and open descriptors in the java process may grow very high:

# ls -l /proc/`pgrep -f java`/fd | wc -l 
76142

Solution

Possible solutions:

  1. Revisit the decision to disable compaction in learner's database
  2. During reconnect, we are in continuous flushing mode (even if working with multiple VirtualMaps concurrently). We could continue flushing a VirtualMap into the same set of files and close them once reconnect is finished.

Alternatives

No response

@OlegMazurov OlegMazurov added Performance Issues related to performance concerns. Platform Virtual Map labels Dec 13, 2024
@poulok poulok added the Platform Tickets pertaining to the platform label Dec 16, 2024
@OlegMazurov
Copy link
Contributor Author

See also #9588

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Performance Issues related to performance concerns. Platform Data Structures Platform Reconnect Platform Virtual Map Platform Tickets pertaining to the platform
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants