The Cost of Over‑Broad Retention: How a 30‑Day Backup Policy Exposed 2 TB of PHI -

A 30‑day retention window is a default in many backup scripts and commercial tools. It feels safe: “I’ll keep the last month of data, that’s enough for recovery.” In 2025, a mid‑size clinic used exactly that policy on a set of encrypted LVM snapshots stored on a single NAS. When the NAS firmware update rolled out, the snapshots were corrupted, and the backup system fell back to a restore‑all mode. The result? 2 TB of unencrypted PHI leaked to a rogue process that had gained write access to the backup directory.

The incident is a textbook example of how a seemingly harmless policy can become a data‑exposure vector. Below is a walk‑through of what happened, why it mattered, and how you can avoid a similar fate in your own environment.

1. The Policy in Context

Element	Default / Common Setting	Why it looks safe
Retention	30 days	“One month is enough to roll back a mistake.”
Storage	Single NAS, local mount	Simpler management, fewer network hops
Encryption	LUKS on the volume, no per‑file encryption	Transparent to backup software
Access	Root‑only, no ACLs	Least privilege at first glance

The policy was baked into a cron job that ran btrfs send/receive every night. The cron line looked like this:

0 2 * * * /usr/local/bin/backup.sh

backup.sh performed a full snapshot, sent it to /mnt/backup, and then pruned snapshots older than 30 days with btrfs subvolume delete. The script had no audit trail beyond the system journal.

2. The Trigger Event

On 2025‑11‑12, the NAS firmware update introduced a bug that caused the btrfs kernel module to mis‑report snapshot sizes. The backup daemon, unaware of the bug, attempted to restore the latest snapshot after the update failed. The restore logic was a simple btrfs receive without integrity checks, so it silently overwrote the existing snapshots.

Because the backup directory was owned by root and had 700 permissions, the only way to read the data was to run as root. A mis‑configured cron job on the same host had a sudo entry that allowed the backup user to run rsync as root without a password. An attacker who compromised the backup user’s credentials could thus pull the entire 2 TB of PHI to a remote server.

3. Why 30 Days Was the Culprit

The 30‑day window meant that the snapshots were stored in a single location for a month. When the restore failed, the entire set of snapshots became a single point of failure. If the retention window had been 7 days, the attacker would have had to compromise the system twice: once to get the 7‑day snapshot and again after the 30‑day snapshot was purged. With a 30‑day window, the attacker had a single window of opportunity.

Moreover, the policy didn’t enforce encryption at rest for each snapshot. The LUKS wrapper was on the entire volume, but the snapshots were still readable by any process that could mount the volume. A more granular approach—encrypting each snapshot with a unique key—would have limited the damage to a single snapshot.

4. Practical Mitigations

4.1. Shorten the Retention Window

A 7‑day window is a good starting point for most production systems. If you need longer retention for compliance, consider a tiered approach:

# Daily snapshots for 7 days
btrfs subvolume snapshot /var/lib/mysql /mnt/backup/daily/$(date +%Y-%m-%d)

# Weekly snapshots for 12 weeks
if [ $(date +%u) -eq 7 ]; then
    btrfs subvolume snapshot /var/lib/mysql /mnt/backup/weekly/$(date +%Y-%W)
fi

This way, a single compromise only exposes a single day or week of data.

4.2. Encrypt Snapshots Individually

Use cryptsetup to create a per‑snapshot LUKS container:

SNAP=$(date +%Y-%m-%d)
btrfs subvolume snapshot /var/lib/mysql /mnt/backup/$SNAP
dd if=/dev/zero of=/mnt/backup/$SNAP/crypt.key bs=1M count=1
cryptsetup luksFormat /mnt/backup/$SNAP/crypt.key /mnt/backup/$SNAP/crypt.img
cryptsetup open /mnt/backup/$SNAP/crypt.img snap_$SNAP
mount /dev/mapper/snap_$SNAP /mnt/backup/$SNAP/mnt

Store the keys in a secure HSM or a vault like HashiCorp Vault. This adds a small overhead but dramatically reduces the blast radius.

4.3. Harden Access Controls

Use ACLs to restrict who can read the backup directory:

setfacl -m u:backup:r /mnt/backup
setfacl -d -m u:backup:r /mnt/backup

Remove sudo passwordless entries that allow backup users to run privileged commands. If you need to run rsync as root, wrap it in a script that checks for a specific environment variable or token.

4.4. Add Integrity Checks

Before restoring a snapshot, verify its integrity:

btrfs check --repair /mnt/backup/$SNAP

Or use btrfs send --checksum and verify the checksum on the receiving side. This prevents silent corruption from propagating.

4.5. Separate Backup Storage

Store backups on a different physical device or in a different rack. If the NAS is compromised, the backups remain safe. Use a dedicated backup server with a hardened kernel and minimal services.

5. Trade‑offs to Consider

Mitigation	Pros	Cons
Shorter retention	Reduces exposure window	May not satisfy regulatory requirements
Per‑snapshot encryption	Limits blast radius	Extra CPU, key management overhead
ACLs & stricter sudo	Tightens access	Requires careful policy management
Integrity checks	Detects corruption early	Adds latency to restore operations
Separate storage	Physical isolation	Higher cost, more maintenance

In practice, a balanced approach works best: keep a 7‑day daily snapshot set, a 12‑week weekly set, encrypt each snapshot, and store them on a separate, hardened device. That gives you quick recovery for most incidents while keeping the risk surface low.

6. Real‑World Troubleshooting

6.1. Snapshot Corruption Detection

If you notice that a restore fails, run:

btrfs check --repair /mnt/backup/$SNAP

If btrfs reports “checksum mismatch,” the snapshot is corrupted. In that case, revert to the previous snapshot or use btrfs send --compress to rebuild.

6.2. Key Retrieval Failures

If the backup server can’t open a snapshot, check the key file permissions:

ls -l /mnt/backup/$SNAP/crypt.key

It should be -rw------- and owned by root. If the key is missing, you’ll need to restore it from your vault.

6.3. Permission Escalation

If a non‑root user can read the backup directory, audit the ACLs:

getfacl /mnt/backup

Look for any entries that grant r to unintended users. Remove them with setfacl -x.

7. Lessons Learned

Retention is not a silver bullet. A 30‑day window can create a single, large attack surface.
Encryption at rest must be granular. Whole‑volume encryption is convenient but insufficient for backups.
Access controls matter. Even if data is encrypted, a mis‑configured sudo entry can expose it.
Integrity checks are cheap. They prevent silent corruption from becoming a data breach.
Physical isolation is still valuable. Separate backup storage reduces the risk of a single point of failure.

By applying these practices, you can keep your backups both useful and secure. The cost of a 30‑day policy isn’t just the 2 TB of PHI that leaked; it’s the time spent investigating, the trust lost with patients, and the potential regulatory penalties. A thoughtful, layered approach to backup retention and hardening pays dividends in both security and operational resilience.