Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid segment file name wrap around. #495

Closed
kjnilsson opened this issue Jan 2, 2025 · 0 comments · Fixed by #496
Closed

Avoid segment file name wrap around. #495

kjnilsson opened this issue Jan 2, 2025 · 0 comments · Fixed by #496
Labels
Milestone

Comments

@kjnilsson
Copy link
Contributor

kjnilsson commented Jan 2, 2025

Describe the bug

segment filenames are limited to 8 characters and are currently a base 10 number that is incremented every time a new segment is created. Although sufficient for most use cases it does mean that for very high throughput Ra clusters it is possible that the number will wrap around after a few years when creating a new segment every second or so.

A wrap around event would cause the segments to be sorted in the wrong order and would cause incorrect log reads.

Expected behavior

by adding a prefix to the segment name we can ensure it is much harder to cause a wrap around event.

Alternative 1: Add the expansion of "~YYMMDD" to the segment name as a prefix. This would allow the creation of over 1000 segment files per second which is extremely unlikely. This approach would also retain the sorting requirements even for servers with pre-existing segment files with the old format.

This alternative creates quite odd looking file names

Alternative 2:

upgrade to a 16 or 20 character name and rename existing segments. this can be done in the segment writer's init function and a marker file can be written to avoid re-entering the upgrade flow once done.

This also should be done for the WAL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant