When a Btrfs Subvolume Crashes After an Unexpected Reboot
A sudden power loss or kernel panic can leave a Btrfs filesystem looking fine on the surface while hiding corruption inside a subvolume. The root of the issue is that Btrfs’ copy‑on‑write guarantees only hold if the metadata journal gets flushed. If the machine dies before that flush, the subvolume can become inconsistent and btrfs check will start complaining. The silver lining is that Btrfs ships with a handful of tools that, when used the right way, can recover most of the data. Below is a practical walk‑through that covers:
- spotting the damaged subvolume
- taking a snapshot before you touch anything
- running
btrfs check --repairsafely - rebuilding the subvolume from the snapshot
- verifying the recovered data
- a quick look at the security‑related trade‑offs (autodefrag, noatime, sync, etc.)
The steps assume a recent kernel (≥ 5.15), Btrfs 5.15‑stable, and root access. All commands are run as root unless noted.
1. Spotting the Problem
You’ll usually see “filesystem is not clean” at boot or a btrfs error in dmesg. First, confirm that a subvolume is actually damaged.
# Mount read‑only to avoid writing on a broken layout
mount -o ro /dev/nvme0n1p2 /mnt/recovery
# List subvolumes and look for status markers
btrfs subvolume list /mnt/recovery | grep -E '^[0-9]+'
If a subvolume shows corrupt or unmounted, that’s the suspect. A quick btrfs check --repair on the device can confirm:
btrfs check --repair /dev/nvme0n1p2
If you see ERROR: Inconsistent subvolume, you’re dealing with a damaged subvolume.
Tip: Always start with a read‑only mount. Writing to a broken subvolume can make recovery harder.
2. Snapshot the Whole Filesystem
Before you start any repair work, make a read‑only snapshot of the entire filesystem. It gives you a point‑in‑time copy to fall back on.
# Read‑only snapshot named pre‑repair
btrfs subvolume snapshot -r /mnt/recovery /mnt/recovery/pre-repair
If you have a spare disk, copy the snapshot out for extra safety:
rsync -aAXHAX --progress /mnt/recovery/pre-repair/ /mnt/backup/pre-repair/
3. Running btrfs check --repair
btrfs check is the go‑to tool for fixing metadata, but it can be destructive if misused. The safest bet is to run it on the unmounted device, not a mounted filesystem.
# Unmount first
umount /mnt/recovery
# Run repair
btrfs check --repair /dev/nvme0n1p2
What to watch for:
ERROR: Inconsistent subvolume– subvolume metadata out of sync.ERROR: Corrupt chunk– might mean a full rebuild is needed.
If btrfs check reports a successful repair, remount and test. If it fails or the subvolume is still corrupted, move to the next step.
4. Rebuilding the Subvolume from the Snapshot
When the subvolume is irreparably damaged, rebuild it from the snapshot you took earlier.
# Mount the snapshot read‑only
mount -o ro /dev/nvme0n1p2 /mnt/recovery/pre-repair
# Create a fresh subvolume to replace the bad one
btrfs subvolume create /mnt/recovery/new_subvol
# Copy data over
rsync -aAXHAX --progress /mnt/recovery/pre-repair/ /mnt/recovery/new_subvol/
After copying, delete the damaged subvolume and rename the new one:
btrfs subvolume delete /mnt/recovery/damaged_subvol
mv /mnt/recovery/new_subvol /mnt/recovery/damaged_subvol
Remount the filesystem:
mount /dev/nvme0n1p2 /mnt/recovery
5. Verifying Integrity
Once the subvolume is back, make sure the data is intact.
# Quick checksum summary
btrfs filesystem df /mnt/recovery
# Run a full scrub
btrfs scrub start -B -c 1 /mnt/recovery
btrfs scrub status /mnt/recovery
The scrub will read every block and verify its checksum. If any errors pop up, investigate further. In most cases, the snapshot copy will be clean because it was taken before the crash.
6. Security‑Related Trade‑offs
6.1. Disable autodefrag on Sensitive Subvolumes
autodefrag can help with write‑heavy workloads but also writes metadata more aggressively. If you’re protecting sensitive data, disabling it reduces the risk of corruption during a sudden power loss.
btrfs property set /mnt/recovery/damaged_subvol autodefrag no
It hurts performance a bit but keeps the metadata steadier.
6.2. Use noatime / nodiratime
Mounting with noatime and nodiratime cuts down on write traffic, which in turn lowers the chance of corruption. It also keeps file‑access times from leaking.
mount -o remount,noatime,nodiratime /mnt/recovery
6.3. Enable sync on Critical Mounts
For logs or audit data, you can force an fsync on every write by adding the sync option. It guarantees that data is flushed before the system acknowledges the write, at the cost of noticeable performance loss.
mount -o remount,sync /mnt/recovery
7. Common Pitfalls and How to Avoid Them
| Pitfall | Why It Happens | Fix |
|---|---|---|
Running btrfs check --repair on a mounted filesystem | The tool expects an unmounted device | Unmount first, or avoid --force on a live system |
| Forgetting to snapshot | You lose the last known good state | Always snapshot before repair |
Using rsync without -a | Permissions/ACLs may be lost | Use -aAXHAX to preserve everything |
| Skipping the scrub | Corruption may still exist | Run btrfs scrub after restoration |
Ignoring btrfs check warnings | Minor warnings can hint at deeper problems | Treat all warnings seriously |
8. Automating the Process
In production, script the workflow and run it nightly. Here’s a minimal Bash script that does the whole cycle:
#!/usr/bin/env bash
set -euo pipefail
DEVICE="/dev/nvme0n1p2"
MOUNTPOINT="/mnt/recovery"
SNAP="/mnt/recovery/pre-repair"
# 1. Snapshot
btrfs subvolume snapshot -r "$MOUNTPOINT" "$SNAP"
# 2. Unmount
umount "$MOUNTPOINT"
# 3. Repair
btrfs check --repair "$DEVICE"
# 4. Remount
mount "$DEVICE" "$MOUNTPOINT"
# 5. Scrub
btrfs scrub start -B -c 1 "$MOUNTPOINT"
btrfs scrub status "$MOUNTPOINT"
Schedule it with cron:
0 3 * * * /usr/local/bin/btrfs-repair.sh >> /var/log/btrfs-repair.log 2>&1
9. When to Seek Professional Help
If btrfs check tells you it can’t repair a chunk or the filesystem is in an unrecoverable state, it’s time to consider:
- Restoring from a backup
- Using a snapshot taken by another tool (e.g., Timeshift)
- Engaging a data‑recovery specialist
At that point, the risk of data loss outweighs the effort of a DIY repair.
10. Final Checklist
- Snapshot the entire filesystem before you touch anything.
- Unmount the filesystem; never run repair on a mounted device.
- Run
btrfs check --repairand review the output. - If the subvolume remains corrupted, rebuild it from the snapshot.
- Remount and run a scrub to verify integrity.
- Apply security‑centric mount options (
noatime,nodiratime,sync) as needed. - Automate the routine for regular health checks.
Follow this workflow and you’ll have a good chance of recovering a Btrfs subvolume after an unexpected reboot while keeping your system hardened against future incidents.
See also
- Application-Level Firewalls
- When /etc/hosts Won’t Resolve: A Quick Systemd‑Resolved Fix
- Recovering a Kernel Panic After Adding a New Initramfs Module
- Multi‑jump bastion access without storing keys: practical ssh‑agent forwarding
- Rolling Back a Broken Flatpak Update
- Fixing systemd-resolved Ignoring /etc/hosts