Taming Log Rotation: Strategies for Preventing /var/log Overflow on Busy Systems -

Introduction to Log Rotation

I’ve seen log files grow out of control and bring down entire systems, so log rotation is something I take seriously. It’s a crucial aspect of Linux system maintenance, ensuring that log files don’t consume all available disk space. On busy systems, logs can fill up quickly, leading to issues with system performance and even causing services to fail.

Understanding Log Rotation

Log rotation involves periodically switching out log files, typically by renaming the current log file and starting a new one. This process can be configured to occur at set intervals, such as daily or weekly, and can also be triggered by the size of the log file. Most Linux distributions come with a log rotation system pre-configured, often using the logrotate utility. Don’t bother with manual log rotation unless you have a specific reason to do so - logrotate is a well-tested and reliable tool.

Configuring Log Rotation

The logrotate utility is typically configured using the /etc/logrotate.conf file and additional configuration files in the /etc/logrotate.d/ directory. These files specify which log files to rotate, how often to rotate them, and what to do with the rotated logs. For example, the following configuration file rotates the /var/log/syslog file daily and keeps the last 7 days’ worth of logs:

/var/log/syslog {
    daily
    missingok
    notifempty
    delaycompress
    compress
    maxsize 100M
    maxage 7
    postrotate
        invoke-rc.d rsyslog rotate > /dev/null
    endscript
}

This configuration file specifies that the /var/log/syslog file should be rotated daily, and that the rotated logs should be compressed and kept for a maximum of 7 days. I usually start with a simple configuration like this and adjust as needed.

Strategies for Preventing Log Overflow

There are several strategies that can be used to prevent log overflow on busy systems:

1. Increase Disk Space

One of the simplest ways to prevent log overflow is to increase the amount of disk space available for log files. This can be done by adding additional storage devices or by increasing the size of the existing storage devices. In practice, this is often the most straightforward solution.

2. Configure Log Rotation to Occur More Frequently

Configuring log rotation to occur more frequently can help prevent log overflow by ensuring that log files do not have a chance to grow too large. For example, rotating logs every 4 hours instead of daily can help prevent overflow.

3. Use a Separate Partition for Log Files

Using a separate partition for log files can help prevent log overflow by ensuring that log files do not consume all available disk space on the system. This can be especially useful on systems where log files are very large or where disk space is limited.

4. Monitor Log File Size

Monitoring log file size can help prevent log overflow by alerting system administrators when log files are approaching their maximum size. This can be done using tools such as logwatch or munin. The real trick is to set up monitoring that’s effective but not overwhelming.

Security Considerations

From a security perspective, it’s especially important to ensure that log files are properly rotated and stored. This can help prevent attackers from using log files to gain information about the system or to identify vulnerabilities. Additionally, ensuring that log files are properly secured can help prevent unauthorized access to sensitive information. I’ve seen this go wrong when log files are not properly secured, and it’s not a pleasant experience.

For more information on log rotation and security, see the systemd documentation on journal configuration and the Debian documentation on log rotation.

Troubleshooting Log Rotation Issues

If log rotation is not working as expected, there are several things that can be checked:

1. Check the Log Rotation Configuration

The first thing to check is the log rotation configuration. Ensure that the configuration files are correct and that the logrotate utility is configured to rotate the correct log files.

2. Check the System Logs

The next thing to check is the system logs. Look for any error messages related to log rotation, such as permission errors or errors related to the logrotate utility.

3. Check the Disk Space

Finally, check the disk space available for log files. If the disk is full, log rotation may not be working correctly. This is where people usually get burned - they forget to check the basics.

Best Practices

By configuring log rotation to occur more frequently, using a separate partition for log files, monitoring log file size, and ensuring that log files are properly secured, system administrators can help prevent log overflow and ensure that their systems remain secure and stable. For more information on log rotation and system maintenance, see the Red Hat documentation on system administration.