Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: avoid panic when the file is corrupted in disk cache #1130

Merged
merged 9 commits into from
Aug 8, 2023

Conversation

ShiKaiWi
Copy link
Member

@ShiKaiWi ShiKaiWi commented Aug 3, 2023

Rationale

  • The corrupted file in the disk cache will lead to the panic of the server
  • The whole cache page file is fetched when only a range of the file is accessed, leading to high memory consumption

Detailed Changes

  • Store the file size in the memory for file integrity check
  • Omit the corrupted file in the disk cache rather than panic
  • Do IO operation outside the meta data cache lock
  • Read only the involved range of a cached paged file instead of the whole file

Test Plan

Existing tests and a newly designed test.

@ShiKaiWi ShiKaiWi force-pushed the fix-avoid-massive-backtrace-output branch 4 times, most recently from 7da4af9 to ee8a5f3 Compare August 4, 2023 09:21
@ShiKaiWi ShiKaiWi force-pushed the fix-avoid-massive-backtrace-output branch 2 times, most recently from 0ad7674 to 3d1ef7b Compare August 7, 2023 08:37
@ShiKaiWi ShiKaiWi force-pushed the fix-avoid-massive-backtrace-output branch from 3d1ef7b to 3670f22 Compare August 8, 2023 06:14
components/object_store/src/disk_cache.rs Show resolved Hide resolved
components/object_store/src/disk_cache.rs Outdated Show resolved Hide resolved
components/object_store/src/disk_cache.rs Outdated Show resolved Hide resolved
components/object_store/src/disk_cache.rs Outdated Show resolved Hide resolved
components/object_store/src/disk_cache.rs Show resolved Hide resolved
components/object_store/src/disk_cache.rs Outdated Show resolved Hide resolved
components/object_store/src/disk_cache.rs Outdated Show resolved Hide resolved
components/object_store/src/disk_cache.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@jiacai2050 jiacai2050 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ShiKaiWi ShiKaiWi merged commit f5b2175 into apache:main Aug 8, 2023
@ShiKaiWi ShiKaiWi deleted the fix-avoid-massive-backtrace-output branch August 8, 2023 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants