Using systemd to Manage and Rotate Logs for Forgotten System Services -

Introduction to Log Management with systemd

I’ve seen many Linux admins struggle with log management, especially when it comes to system services that are often overlooked. Systemd is a powerful system and service manager that provides a wide range of features, including process management, dependency handling, and log management. In this article, I’ll focus on using systemd to manage and rotate logs for system services.

Understanding systemd Logs

The real trick is to understand how systemd logs work. Systemd logs are stored in a binary format, which can be read using the journalctl command. This command provides a powerful way to filter, search, and manage system logs. By default, systemd stores logs in /var/log/journal, but this can be configured to use a different location. Don’t bother with trying to read the binary logs directly - just use journalctl.

To view the current log entries, you can use the following command:

journalctl -u <service_name>

Replace <service_name> with the name of the service you want to view logs for. For example, to view logs for the ssh service, you would use:

journalctl -u ssh

In practice, I usually start with a simple journalctl command to get a feel for the logs.

Configuring Log Rotation

This is where people usually get burned - log rotation. Systemd provides a built-in log rotation mechanism, which can be configured using the journald.conf file. This file is usually located in /etc/systemd/journald.conf. To configure log rotation, you can add the following lines to the file:

SystemMaxUse=100M
SystemKeepFree=20M

The first line sets the maximum size of the log files to 100M, while the second line sets the amount of free space to keep on the disk. You can adjust these values to suit your needs. I usually set the maximum size to a reasonable value to prevent log files from growing too large.

Managing Forgotten System Services

Forgotten system services can include services that are not regularly monitored or maintained. These services can include things like cron, rsyslog, or auditd. To manage logs for these services, you can use the journalctl command with the -u option, followed by the name of the service.

For example, to view logs for the cron service, you would use:

journalctl -u cron

You can also use the --since and --until options to filter logs by time. For example:

journalctl -u cron --since=yesterday --until=1hourago

This command will show logs for the cron service from yesterday to 1 hour ago.

Security Considerations

When managing logs, security is key. Logs can contain sensitive information, such as user credentials or encryption keys. To protect logs, you can use encryption or access controls.

Systemd provides a feature called “log sealing”, which allows you to encrypt logs using a seal key. To enable log sealing, you can add the following line to the journald.conf file:

Seal=true

You can also use access controls to restrict access to logs. For example, you can use sudo to restrict access to the journalctl command.

Troubleshooting Log Issues

If you encounter issues with logs, you can use the journalctl command with the --verbose option to get more detailed output. For example:

journalctl -u <service_name> --verbose

You can also use the --debug option to get even more detailed output.

Best Practices for Log Management

To get the most out of systemd log management, it’s essential to follow best practices. Here are some tips:

Regularly review logs to detect potential issues
Use log rotation to prevent log files from growing too large
Use encryption or access controls to protect sensitive information
Use journalctl to filter and search logs
Use journald.conf to configure log settings

For more information on systemd log management, you can visit the systemd.io website.