-
Notifications
You must be signed in to change notification settings - Fork 15
Switch to Soft Quota
At the moment, containers are created with a btrfs hard quota. This is not desirable for a number reasons:
- When quota run out, the container ends up being unrecoverable for the end user:
root@Container-1-JBe:/# rm test.dat
rm: cannot remove 'test.dat': Disk quota exceeded
- Quotas on btrfs is experimental at the very best. From the "btrfs quota" man page:
STABILITY STATUS
The qgroup implementation has turned out to be quite difficult as it
affects the core of the filesystem operation. The users have hit
various corner cases over time, eg. wrong accounting or system
instability. The situation is gradually improving but currently (4.7)
there are still issues found and fixed.
We have experienced occasional failures where btrfs became read-only. While the exact cause of this has not yet been determined it is strongly suspected that it could also be related to quotas.
-
Keep btrfs hard quotas but set them to a size 2-3 * the desired container size (just to catch run-away containers)
-
Implement two timed checks running every X hours (X need to be agreed upon but shouldn't need to run often - 6 hours is probably about right) which will perform the following checks:
-
Check if container usage is > 0.9 * container quota - if so - send warning
-
Check if container usage is > 1.5 * container quota - if so - stop container
-
This way, the owner of the container will receive a warning when he's getting close to having used his allocated space, but he will have plenty of time to clean up or migrate to a larger container. Also if container end up stopped, the user can restart the container and recover (either by cleaning up or migrating).
For performance reasons the btrfs tool should be used to check:
sudo btrfs fi df /var/snap/subutai-master/common/lxc/<container>/
Data, single: total=2.01GiB, used=543.50MiB
System, DUP: total=8.00MiB, used=16.00KiB
Metadata, DUP: total=1.00GiB, used=48.19MiB
GlobalReserve, single: total=16.00MiB, used=0.00B
Related issue for comments https://github.com/subutai-io/base/issues/2226