Strategies on avoiding btrfs failures

[happyaron]

Since last week we have concluded a series of strategies on avoiding btrfs failures, using this issue as a reference to keep track of all the ideas and TODO items.

We suspect btrfs is vulnerable when it is not cleanly umounted, as well as it tends to fail under heavy workload (overload) with quota in use. To avoid these problems as much as possible, we want to do clean umount when the system is working normally, and run defensive maintenance tools both regularly and on failure happens.

1. Determine what has been triggering the btrfs failures, so that a proper kernel bug report can be made. This is tracked in Btrfs Fail #229
2. Detect unclean shutdowns, so that we are able to run btrfs maintenance before mounting the file system. This is tracked in Detect unclean shutdown #232
3. Run btrfs scrub at least monthly to maintain the file system integrity.
4. Agent should stop all containers on shutdown and removal. This is tracked in Agent should stop all containers on peer reboot/shutdown agent#270
5. Set a higher btrfs quota for purely defensive purpose, and let hub/agent do the daily space quota enforcement. This is tracked in Run disk quota excess check  peer-os#2226 and Disable btrfs quotas in btrfsinit script #233
6. Reduce excessive data collection from agent, to avoid high sys% performance penalty.

[dilshat]

https://askubuntu.com/questions/464074/ubuntu-thinks-btrfs-disk-is-full-but-its-not