When Disk Space Disappears: Tracking Down and Preventing Logs from Filling Up Your Linux Disks -

Introduction to Disk Space Issues

I’ve seen this go wrong when managing Linux systems - the sudden disappearance of disk space. It’s often caused by log files filling up the disk, large files being stored in unexpected locations, or even malware consuming disk space. In this article, I’ll focus on tracking down and preventing logs from filling up your Linux disks.

Understanding Log Files

Log files are essential for Linux system administration, providing valuable information about system events, errors, and security issues. However, if not properly managed, log files can grow rapidly and consume large amounts of disk space. The most common log files that can cause issues are:

/var/log/syslog
/var/log/messages
/var/log/secure
/var/log/auth.log

These log files are typically managed by the system logging daemon, such as rsyslog or systemd-journald. To prevent log files from filling up the disk, it’s essential to configure the logging daemon to rotate and compress log files regularly. Don’t bother with manual rotations - it’s better to automate the process.

Configuring Log Rotation

The real trick is to configure log rotation using the logrotate utility, which is typically configured to run daily or weekly. To do this, you can edit the /etc/logrotate.conf file and add the following lines:

/var/log/syslog {
    daily
    missingok
    notifempty
    delaycompress
    compress
    maxsize 10M
    maxage 7
    postrotate
        invoke-rc.d rsyslog reload > /dev/null
    endscript
}

This configuration will rotate the /var/log/syslog file daily, compressing it and keeping the last 7 days of logs. In practice, you may need to adjust the rotation schedule and log file size limits based on your system’s specific needs.

Using systemd-journald

If you’re using a systemd-based system, you can use systemd-journald to manage log files. systemd-journald provides a more efficient and flexible way of managing log files, allowing you to configure log rotation, compression, and retention policies. To configure systemd-journald, you can edit the /etc/systemd/journald.conf file and add the following lines:

[Journal]
SystemMaxUse=100M
SystemKeepFree=20M
SystemMaxFileSize=10M

This configuration will limit the total size of the journal to 100M, keep at least 20M of free space, and limit the size of individual log files to 10M.

Monitoring Disk Space

To prevent disk space issues, it’s essential to monitor disk space regularly. I usually start with the df command to check disk space usage:

df -h

This will display the disk space usage for each mounted filesystem. You can also use the du command to check the size of specific directories:

du -sh /var/log

This will display the total size of the /var/log directory.

Security Considerations

When managing log files, it’s essential to consider security implications. Log files can contain sensitive information, such as passwords, IP addresses, and system configuration details. This is where people usually get burned - failing to secure their log files can lead to serious security breaches. To minimize security risks, it’s recommended to:

Use secure logging protocols, such as TLS or SSH, to transmit log files to remote servers.
Configure log files to be owned by a specific user and group, with restricted permissions.
Use encryption to protect log files, especially when storing them on remote servers.

For more information on secure logging practices, you can refer to the systemd-journald documentation or the rsyslog documentation.

Troubleshooting

If you’re experiencing disk space issues due to log files, you can try the following troubleshooting steps:

Check the log file size using the du command.
Check the log rotation configuration using the logrotate utility.
Check the systemd-journald configuration using the systemd-journald command.
Use the journalctl command to check the journal size and configuration.