Taming Disk-Hungry Logs with systemd-journald and logrotate -

Introduction to Log Management

I’ve seen log files grow out of control and consume disk space, affecting system performance. To tame disk-hungry logs, I recommend using systemd-journald and logrotate. These tools help manage log data, making it easier to troubleshoot, debug, and perform security audits.

Understanding systemd-journald

systemd-journald is a system service that collects and stores log messages from various sources. It provides a centralized logging system, which I find more efficient than traditional text-based log files. To view log messages, use the journalctl command:

journalctl

You can also filter log messages based on priority, timestamp, or specific services:

journalctl -u sshd -p err

This is where people usually get burned - they don’t realize how much data systemd-journald can store. Don’t bother with trying to manage it manually; instead, use journalctl to get the job done.

Configuring systemd-journald

By default, systemd-journald stores log messages in /var/log/journal/. You can configure the storage location and other settings by editing the /etc/systemd/journald.conf file. For example, to set the maximum disk space used by systemd-journald, add the following line:

SystemMaxUse=100M

This will limit the disk space used by systemd-journald to 100MB. I usually start with a smaller value and adjust as needed to avoid running out of disk space.

Introduction to logrotate

logrotate is a utility that helps manage log files by rotating, compressing, and deleting them. It’s commonly used to manage traditional text-based log files, but it can also be used with systemd-journald. logrotate is typically run as a daily cron job, which checks for log files that need to be rotated and performs the necessary actions.

To configure logrotate, create a configuration file in /etc/logrotate.d/. For example, to rotate the syslog log file, create a file called syslog with the following contents:

/var/log/syslog {
    daily
    missingok
    notifempty
    delaycompress
    compress
    maxsize 100M
    maxage 7
    postrotate
        /usr/sbin/service rsyslog restart > /dev/null
    endscript
}

The real trick is to find the right balance between log retention and disk space usage. In practice, I’ve found that rotating logs daily and keeping them for a week is a good starting point.

Using logrotate with systemd-journald

To use logrotate with systemd-journald, configure logrotate to rotate the systemd-journald log files. Create a configuration file in /etc/logrotate.d/ called journald with the following contents:

/var/log/journal {
    daily
    missingok
    notifempty
    delaycompress
    compress
    maxsize 100M
    maxage 7
    postrotate
        /usr/bin/journalctl --vacuum-size=100M
    endscript
}

This configuration file tells logrotate to rotate the systemd-journald log files daily, compressing and deleting old log files as needed.

Troubleshooting Log Issues

When troubleshooting log issues, check the log files for errors and warnings. Use the journalctl command to view log messages and filter them based on priority or timestamp:

journalctl -p err -S "1 hour ago"

You can also use the logrotate command to check the status of log rotation:

logrotate -f /etc/logrotate.d/syslog

I’ve seen this go wrong when log files are not properly rotated, causing disk space issues.

Security Considerations

When managing log files, consider security implications. Log files can contain sensitive information, such as user credentials or encryption keys. To mitigate these risks, ensure that log files are stored securely and access is restricted to authorized personnel. You can use tools like logrotate to compress and encrypt log files, making it more difficult for unauthorized access.

For more information on systemd-journald and logrotate, refer to the systemd.io and github.com/logrotate websites.