Skip to content

Commit

Permalink
fix: avoid panic when the file is corrupted in disk cache (#1130)
Browse files Browse the repository at this point in the history
## Rationale
- The corrupted file in the disk cache will lead to the panic of the
server
- The whole cache page file is fetched when only a range of the file is
accessed, leading to high memory consumption

## Detailed Changes
- Store the file size in the memory for file integrity check
- Omit the corrupted file in the disk cache rather than panic
- Do IO operation outside the meta data cache lock
- Read only the involved range of a cached paged file instead of the
whole file

## Test Plan
Existing tests and a newly designed test.
  • Loading branch information
ShiKaiWi authored Aug 8, 2023
1 parent b69f82d commit f5b2175
Show file tree
Hide file tree
Showing 2 changed files with 506 additions and 281 deletions.
11 changes: 6 additions & 5 deletions analytic_engine/src/setup.rs
Original file line number Diff line number Diff line change
Expand Up @@ -519,12 +519,18 @@ fn open_storage(
}
};

store = Arc::new(StoreWithMetrics::new(
store,
engine_runtimes.io_runtime.clone(),
));

if opts.disk_cache_capacity.as_byte() > 0 {
let path = Path::new(&opts.disk_cache_dir).join(DISK_CACHE_DIR_NAME);
tokio::fs::create_dir_all(&path).await.context(CreateDir {
path: path.to_string_lossy().into_owned(),
})?;

// TODO: Consider the readonly cache.
store = Arc::new(
DiskCacheStore::try_new(
path.to_string_lossy().into_owned(),
Expand All @@ -538,11 +544,6 @@ fn open_storage(
) as _;
}

store = Arc::new(StoreWithMetrics::new(
store,
engine_runtimes.io_runtime.clone(),
));

if opts.mem_cache_capacity.as_byte() > 0 {
let mem_cache = Arc::new(
MemCache::try_new(
Expand Down
Loading

0 comments on commit f5b2175

Please sign in to comment.