Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Write a cache cleaner for a file system #673

Merged
merged 10 commits into from
Jul 20, 2017
21 changes: 21 additions & 0 deletions efs_cache_cleaner/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# efs_cache_cleaner

This service allows us to purge our EFS cache based on the age and size of items.

Some of our ECS applications use local storage as a cache (for example, Loris).
Because ECS instances have very little disk space, we use an EFS mount rather than the host disk.
However, EFS storage has a price – although it can grow to any size, it would be expensive to let it do so!
This service deletes files from our EFS cache to keep our costs sustainable.

Our caches are bounded by two configurable:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/s/configurable/parameters


* **maximum age** -- if an item is in the cache but hasn't been accessed for more than 30 days, there isn't much value in keeping it in the cache.
We can delete it, and re-fetch it the next time it's requested.

* **maximum size** -- EFS storage is priced based on how much you use.
If the cache exceeds a given size, we can delete items from the cache until it comes back under the limit.
Items are deleted in order of last access time -- items which haven't been accessed very recently are deleted first.

Both of these parameters are configurable.

Although we're using EFS, a lot of the core logic could be repurposed to run against a conventional file system.