Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate how to handle examining files as part of compaction planning #3526

Open
keith-turner opened this issue Jun 22, 2023 · 1 comment
Open

Investigate how to handle examining files as part of compaction planning #3526

keith-turner opened this issue Jun 22, 2023 · 1 comment
Milestone
4.0.0

Comments

@keith-turner
Copy link
Contributor

As part of selecting files to compact for user compactions the compaction selector plugin offered examining summaries and sample data as part of this decision. This data supported use cases like the TooManyDeletesSelector which could trigger a full tablet compaction when the ratio of deletes in a tablet exceeded a certain threshold.

When the examination ran in tablet servers this examination had much more CPU and memory available than what will be available when running the examination in the manager. In #3513 which moved user compaction from tablet server to the manager this functionality was not implemented because of concerns about resources.
@keith-turner keith-turner converted this from a draft issue Jun 22, 2023
@dlmarion dlmarion mentioned this issue Jul 25, 2023
Open
keith-turner added a commit to keith-turner/accumulo that referenced this issue Sep 29, 2023
@keith-turner
Implements missing compacton selection functionality
5b73334 
Compaction selection functionality that opened files was not implemented
because it opens user files in the manager.  This change implements the
functionality inorder to get integration test passing.  There is already
an open issue about finding a better way to do this. See apache#3526
This was referenced Sep 29, 2023
Merged
Open
keith-turner added a commit that referenced this issue Oct 6, 2023
@keith-turner
Implements missing compacton selection functionality (#3796)
164c85c 
Compaction selection functionality that opened files was not implemented
because it opens user files in the manager.  This change implements the
functionality inorder to get integration test passing.  There is already
an open issue about finding a better way to do this. See #3526
This was referenced Oct 10, 2023
Closed
Closed
@keith-turner
Copy link
Contributor Author

One possible way to solve this is via #3822 and #3559 which would allow fate operations that do this examination to run in multiple processes.

@ctubbsii ctubbsii added this to the 4.0.0 milestone Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Accumulo: Elasticity
Status: No status
Milestone
4.0.0
Development

No branches or pull requests
2 participants
@keith-turner @ctubbsii