Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: improve compaction logic #373

Merged
merged 2 commits into from
Jul 9, 2024
Merged

Conversation

tri-adam
Copy link
Member

@tri-adam tri-adam commented Jul 4, 2024

Improve the behaviour of image compaction when adding/deleting data objects.

Improve compaction when OptDeleteCompact is supplied to DeleteObject. Do not fail when compaction is requested on a data object that is not at the end of the data section. When compaction is requested, always calculate the end of the data section based on the in-use descriptors.

Previously, AddObject would simply append the data object to the end of the data section. If one or more object(s) had previously been deleted from the image, this could result in wasted space. Now, we trim the data section based on in-use objects before adding the new object.

Add/update tests to reflect new behaviour.

Closes #193

@tri-adam tri-adam self-assigned this Jul 4, 2024
Copy link

codecov bot commented Jul 4, 2024

Codecov Report

Attention: Patch coverage is 83.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 62.64%. Comparing base (e3aa617) to head (e8dad67).

Files Patch % Lines
pkg/sif/delete.go 57.14% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #373      +/-   ##
==========================================
+ Coverage   62.59%   62.64%   +0.05%     
==========================================
  Files          37       37              
  Lines        3240     3234       -6     
==========================================
- Hits         2028     2026       -2     
+ Misses       1046     1043       -3     
+ Partials      166      165       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tri-adam tri-adam mentioned this pull request Jul 4, 2024
@tri-adam tri-adam marked this pull request as ready for review July 4, 2024 20:55
tri-adam added 2 commits July 8, 2024 15:04
Previously, an add operation would simply append the data object to the
end of the data section. If one or more object(s) had previously been
deleted from the image, this could result in wasted space. Now, we trim
the data section based on in-use objects before adding the new object.
Update tests to reflect this new behaviour.
Improve compaction via OptDeleteCompact. Do not fail when compaction is
requested on a data object that is not at the end of the data section.
When compaction is requested, always calculate the end of the data
section based on the in-use descriptors.
Copy link
Member

@dtrudg dtrudg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks sensible to me, but I'm missing something....

image

What is now causing these to be 3 bytes shorter? I'm clearly missing the cause...

@tri-adam
Copy link
Member Author

tri-adam commented Jul 8, 2024

What is now causing these to be 3 bytes shorter? I'm clearly missing the cause...

That's a result of this line:

https://github.com/tri-adam/sif/blob/e8dad67f88b7af7d7d3d4933d1c301428c17a319/pkg/sif/create.go#L97

Those two test cases delete the last object in the SIF without supplying the compact option, and then add a new object. Previously, the code would append after the end of the data section (which still included the deleted object.) The new code calculates where the end of the valid data actually is, and appends there.

@dtrudg
Copy link
Member

dtrudg commented Jul 9, 2024

What is now causing these to be 3 bytes shorter? I'm clearly missing the cause...

That's a result of this line:

https://github.com/tri-adam/sif/blob/e8dad67f88b7af7d7d3d4933d1c301428c17a319/pkg/sif/create.go#L97

Those two test cases delete the last object in the SIF without supplying the compact option, and then add a new object. Previously, the code would append after the end of the data section (which still included the deleted object.) The new code calculates where the end of the valid data actually is, and appends there.

Thanks... apologies I missed that!

@tri-adam tri-adam merged commit 48f265f into sylabs:main Jul 9, 2024
1 check passed
@tri-adam tri-adam deleted the compaction branch July 9, 2024 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve Image Compaction Logic
2 participants