Freeing up 5 GB of disk space by pruning old journalctl logs on a 20 GB VPS -

Freeing 5 GB of Disk Space by Pruning Old journalctl Logs on a 20 GB VPS

A 20 GB VPS is a tight fit. On a busy host, the system journal can grow fast, especially with frequent reboots, kernel messages, or misconfigured services that spam systemd-journald. When the journal consumes several gigabytes, you hit the “no space left on device” error on /var/log or /var/run, and the OS starts throttling services or, worse, refuses to start new ones.

Below is a step‑by‑step walkthrough that shows how to safely reclaim disk space by pruning old journal entries, while keeping the logs you actually need for compliance and debugging. The techniques work on any systemd‑based distribution (Debian, Ubuntu, CentOS, Fedora, Arch, etc.) and are fully reproducible with the commands shown.

1. Understand What `journalctl` Is Managing

systemd-journald stores logs in a binary format under /var/log/journal/<machine-id>/. The data is compressed and indexed, so the journal is more efficient than plain text logs, but it still consumes disk space. The size is governed by two main configuration knobs:

Setting	Default	Effect
`SystemMaxUse=`	10 % of the filesystem	Max total size of the journal
`SystemMaxFileSize=`	10 MB	Max size of a single journal file

If you have a 20 GB root partition, the default 10 % cap is 2 GB. If you see the journal at 5 GB, you’ve exceeded the default, likely because the cap was overridden or the filesystem is a different size.

2. Quick Check: How Much Is the Journal Using?

# Show the journal directory size
sudo du -sh /var/log/journal/*

# Or, if you want a more granular view
sudo journalctl --disk-usage

Typical output:

$ sudo journalctl --disk-usage
Archived and volatile logs take up 4.9 GB

If the number is higher than expected, you’re ready to prune.

3. Decide What You Need to Keep

Before deleting anything, decide:

Compliance – Do you have to keep logs for a certain period (e.g., 90 days for audit)?
Debugging – Are you troubleshooting a recent issue that might need older entries?
Security – Do you need to retain logs that could help detect intrusion attempts?

If you’re unsure, keep at least the last 30 days. That’s a good compromise between disk usage and forensic value.

4. Configure a Reasonable Retention Policy

Edit /etc/systemd/journald.conf:

sudo nano /etc/systemd/journald.conf

Uncomment and set:

# Keep logs for 30 days
SystemMaxUse=2G
SystemMaxFileSize=200M
SystemMaxRetentionSec=2592000

SystemMaxUse caps the total size.
SystemMaxFileSize limits individual files.
SystemMaxRetentionSec (30 days = 30×24×60×60 = 2,592,000 s).

After editing, reload:

sudo systemctl restart systemd-journald

This prevents the journal from growing unchecked in the future.

5. Prune Old Entries Manually

If you need to free space immediately, use journalctl --vacuum-* options.

5.1 Vacuum by Size

sudo journalctl --vacuum-size=1G

This keeps the newest logs until the total size is ≤ 1 GB. Adjust the target size to leave a buffer for future growth.

5.2 Vacuum by Time

sudo journalctl --vacuum-time=30d

Keeps only the last 30 days. If you want to keep 90 days, replace 30d with 90d.

5.3 Vacuum by Number of Files

sudo journalctl --vacuum-files=50

Keeps the 50 most recent journal files. Useful if you’re more concerned about file count than size.

Tip: Combine options for fine‑tuned control:

sudo journalctl --vacuum-size=1G --vacuum-time=30d

The command will delete the oldest entries until both conditions are met.

6. Verify the Result

After vacuuming, check the size again:

sudo journalctl --disk-usage

You should see a dramatic drop. If you still need more space, repeat the vacuum with a stricter target.

7. Automate Pruning with a Cron Job

To keep the journal from ballooning again, schedule a daily vacuum:

sudo crontab -e

Add:

# Run vacuum every day at 02:00
0 2 * * * /usr/bin/journalctl --vacuum-time=30d >/dev/null 2>&1

This runs silently and keeps the journal under 30 days. Adjust the time to avoid peak usage.

8. Consider Log Rotation for Plain Text Logs

journalctl handles systemd logs, but many services still write to /var/log/*.log. Ensure logrotate is configured:

sudo nano /etc/logrotate.d/syslog

Typical entry:

/var/log/syslog
{
    daily
    rotate 7
    compress
    missingok
    notifempty
    create 0640 root adm
}

This keeps 7 compressed daily logs, freeing space while preserving recent history.

9. Security Implications

Pruning logs is a double‑edged sword:

Action	Risk	Mitigation
Deleting old logs	Loss of forensic evidence	Keep at least 30 days; archive to external storage
Over‑restricting retention	Missing audit trails	Align with compliance requirements
Running vacuum on a busy system	Performance hit	Schedule during low‑traffic windows

Audit Trail – If you’re subject to regulations (PCI‑DSS, HIPAA), check the required retention period. The 30‑day default is often insufficient. In such cases, archive older logs to an immutable storage medium (e.g., a separate encrypted volume or a cloud object store).

Immutable Logs – For critical logs, consider writing them to a separate partition mounted with noatime and nodev flags, and use rsync or scp to copy them to a remote server before vacuuming.

10. Troubleshooting Common Issues

Symptom	Likely Cause	Fix
`journalctl` still shows > 5 GB after vacuum	Journal files are still large; vacuum didn’t delete enough	Increase `--vacuum-size` target or reduce `SystemMaxFileSize`
`systemd-journald` stops writing logs	Disk full before vacuum runs	Add a cron job that runs before the system reaches 90 % usage
`journalctl` errors: “Failed to read journal: Permission denied”	Running as non‑root without `--user`	Use `sudo journalctl` or add user to `systemd-journal` group
`journalctl --disk-usage` shows “0 B” after vacuum	Journal directory is empty	Verify you’re looking at the correct directory; maybe logs are in `/run/log/journal` (volatile)

11. Advanced: Using `systemd-tmpfiles` for Temporary Files

If you’re also dealing with temporary files that accumulate, configure /usr/lib/tmpfiles.d/systemd.conf:

# Clean /tmp and /var/tmp after 10 days
tmpfs /tmp 1777 root root 10d
tmpfs /var/tmp 1777 root root 10d

This ensures that temporary directories don’t fill up the root partition.

12. Summary of Commands

# Check current journal usage
sudo journalctl --disk-usage

# Vacuum to 1 GB
sudo journalctl --vacuum-size=1G

# Vacuum to 30 days
sudo journalctl --vacuum-time=30d

# Vacuum to 50 files
sudo journalctl --vacuum-files=50

# Configure retention in journald.conf
sudo nano /etc/systemd/journald.conf
# (set SystemMaxUse, SystemMaxFileSize, SystemMaxRetentionSec)

# Restart journald
sudo systemctl restart systemd-journald

# Schedule daily vacuum
0 2 * * * /usr/bin/journalctl --vacuum-time=30d >/dev/null 2>&1

13. When to Archive Instead of Delete

If you need to keep logs for longer than the retention policy but still want to free space, move them to an external archive:

# Create an archive directory
sudo mkdir /srv/log-archive
sudo chown root:root /srv/log-archive
sudo chmod 700 /srv/log-archive

# Move old journal files
sudo journalctl --vacuum-time=30d --rotate
sudo mv /var/log/journal/* /srv/log-archive/

You can then compress the archive:

sudo tar -czf /srv/log-archive/journal-$(date +%Y%m%d).tar.gz -C /srv/log-archive .

Store the tarball on an encrypted external drive or a cloud bucket with immutable retention.

14. Final Checklist

Verify current journal size.
Set a sensible retention policy in journald.conf.
Vacuum to the desired size/time.
Confirm space is freed.
Automate vacuum with cron.
Ensure compliance requirements are met.
Archive older logs if needed.

By following these steps, you’ll keep a 20 GB VPS healthy, avoid disk‑full crashes, and maintain a secure, audit‑ready logging environment.

Freeing 5 GB of Disk Space by Pruning Old journalctl Logs on a 20 GB VPS

1. Understand What journalctl Is Managing