Taming Log Rotation in systemd: A Practical Approach to Preventing Disk Bloat -

Introduction to Log Rotation

I’ve seen this go wrong when log files grow out of control, filling up the disk and causing system instability. That’s why log rotation is a crucial aspect of Linux system maintenance. With many Linux distributions, including Debian and Arch Linux, adopting systemd as their default init system, understanding how to manage log rotation in a systemd environment is essential. In practice, this means getting familiar with systemd-journald, the component responsible for collecting and storing log messages.

Understanding systemd-journald

Systemd-journald stores logs in a binary format, which can be queried using the journalctl command. Don’t bother with trying to read the binary files directly - just use journalctl to view and manage log messages. To manage log rotation, you need to configure journald to limit the size of the log files. This involves editing the /etc/systemd/journald.conf file:

sudo nano /etc/systemd/journald.conf

In this file, you can adjust the SystemMaxUse and SystemKeepFree parameters to control the maximum size of the log files and the amount of free space to maintain on the disk. The real trick is finding the right balance between log file size and free space.

Configuring Log Rotation

To configure log rotation, you need to create a configuration file in the /etc/systemd/journald.conf.d/ directory. I usually start with a file called 00-log-rotation.conf with the following contents:

[Journal]
SystemMaxUse=100M
SystemKeepFree=500M

This configuration sets the maximum size of the log files to 100MB and maintains at least 500MB of free space on the disk. You can adjust these values according to your needs, but be careful not to set the log file size too low, or you may miss important log messages.

Using journalctl

The journalctl command is a powerful tool for querying and managing log messages. You can use it to view log messages, filter logs by priority or unit, and even delete old log messages. For example, to view log messages from the ssh service, you can use the following command:

journalctl -u ssh

To delete old log messages, you can use the --vacuum-size option:

journalctl --vacuum-size=100M

This command deletes old log messages until the total size of the log files is reduced to 100MB.

Security Considerations

This is where people usually get burned - setting the log file size too low or too high can have serious security implications. If you set the log file size too low, you may miss important log messages that could indicate a security breach. On the other hand, if you set the log file size too high, you may fill up the disk and cause system instability. A good practice is to set the log file size to a reasonable value, such as 100MB, and monitor the log files regularly for any suspicious activity.

Troubleshooting

If you encounter issues with log rotation, you can check the journald configuration file for errors. You can also use the journalctl command to view log messages and diagnose problems. For example, to view log messages from the journald service, you can use the following command:

journalctl -u systemd-journald

This command displays log messages from the journald service, which can help you diagnose issues with log rotation.

Additional Resources

For more information on systemd-journald and log rotation, you can refer to the systemd documentation and the Debian wiki. These resources provide detailed information on configuring and troubleshooting systemd-journald.