How to Get a Systemd Timer Back on Track After a Kernel Upgrade -

Systemd timers can fall silent after a kernel upgrade

When a new kernel lands, the kernel‑specific modules and initramfs changes can break the ExecStartPre or ExecStart scripts that a timer relies on.
The result? A timer that never fires, or that fires with the wrong environment.

Below is a step‑by‑step guide to diagnose and restore a broken timer, with security‑aware notes and trade‑offs you’ll run into in a production or homelab setup.

1. Identify the symptom

systemctl list-timers --all | grep -i <name>

If the timer shows Next: never, Active: inactive (dead), or Last: failed, the unit is not running.
Check the status of the timer and its service:

systemctl status <timer>.timer
systemctl status <service>.service

A common error after a kernel upgrade is:

Failed to start <service>.service: Unit <service>.service is masked.

Failed to start <service>.service: ExecStartPre failed: exit status 1

2. Quick sanity checks

Check	Command	What to look for
Kernel version	`uname -r`	Does it match the version you expect?
Initramfs	`ls /boot/initramfs-$(uname -r).img`	Is the initramfs present?
Systemd version	`systemctl --version`	Some timers use `OnCalendar=` syntax only available in newer systemd releases.
SELinux/AppArmor	`sestatus` / `aa-status`	A policy change during the upgrade can block the service.

If the kernel is newer than the initramfs, rebuild it:

sudo mkinitcpio -P   # Arch
sudo dracut -f       # RHEL/CentOS

3. Inspect the unit files

systemctl cat <timer>.timer
systemctl cat <service>.service

Look for:

EnvironmentFile pointing to a path that no longer exists.
ExecStartPre that calls a script using a hard‑coded kernel path.
ConditionKernelCommandLine or ConditionKernelVersion that may now be false.

If the unit file references a kernel‑specific binary (e.g., /usr/lib/modules/$(uname -r)/...), replace it with a generic path or add a fallback.

Example patch:

sudo systemctl edit <service>.service

Add:

[Service]
ExecStartPre=/usr/bin/true

This bypasses the failing pre‑check. Use with caution: only do this if you understand why the pre‑check failed.

4. Reload systemd and test

sudo systemctl daemon-reload
sudo systemctl restart <timer>.timer

Verify the timer’s next run:

systemctl list-timers | grep <timer>

If the timer still fails, check the journal:

journalctl -u <service>.service -b

The log will often contain the exact failure, e.g., “Permission denied” or “File not found”.

5. Resolve permission or SELinux/AppArmor issues

Kernel upgrades can trigger new security contexts.
If the service is denied by SELinux:

sudo ausearch -m avc -ts recent | tail

Add a permissive rule:

sudo semanage fcontext -a -t <type> "/path/to/file(/.*)?"
sudo restorecon -Rv /path/to/file

For AppArmor, use:

sudo aa-complain /etc/apparmor.d/<profile>

Only use complain mode temporarily; revert to enforce after the timer works.

6. Handle dependencies on kernel modules

Some timers start services that load modules (e.g., i2c-tools, nvme).
If the module is missing in the new kernel, the service will fail.

sudo modprobe -l | grep <module>

If missing, install the module package for the new kernel:

sudo pacman -S linux$(uname -r | cut -d- -f1)-extra   # Arch
sudo dnf install kernel-modules-$(uname -r)          # RHEL/CentOS

After installing, reload the timer.

7. Verify after reboot

A timer that works immediately after a manual restart may still fail after a reboot if the initramfs or boot loader is misconfigured. Reboot and check:

systemctl list-timers | grep <timer>

If the timer is inactive, examine the boot logs:

journalctl -b | grep -i <timer>

8. Security‑aware trade‑offs

Trade‑off	Description	When to use
Bypass pre‑checks	Temporarily replace failing `ExecStartPre` with `/usr/bin/true`.	When you confirm the pre‑check is no longer necessary.
Disable SELinux/AppArmor complain	Allows the service to run while you investigate.	Short‑term debugging; revert to enforce ASAP.
Use a custom initramfs	Rebuild with the exact modules needed.	If the timer depends on modules not present in the default initramfs.
Keep kernel modules pinned	Install the same kernel version as before.	In environments where stability outweighs new features.

9. Best practices to avoid future timer breakage

Pin critical services to a specific kernel
Use ConditionKernelVersion= in the unit file to prevent the service from starting on an incompatible kernel.
```
[Unit]
ConditionKernelVersion=5.15.*
```
Use OnCalendar= with Persistent=true
Guarantees that missed runs are caught after a reboot.
```
[Timer]
OnCalendar=*-*-* 02:00:00
Persistent=true
```
Keep unit files in /etc/systemd/system/
Overrides from the distribution are preserved during upgrades.
Automate initramfs rebuild
Add a hook to your package manager that runs mkinitcpio -P or dracut -f after any kernel package installation.
Audit journal logs after kernel upgrades
A quick script can flag services that failed to start:
```
journalctl -b -p err | grep -E 'Failed to start|ConditionFailed'
```

10. Quick reference checklist

Step	Command	Purpose
Reload systemd	`systemctl daemon-reload`	Apply unit changes
Restart timer	`systemctl restart <timer>.timer`	Trigger immediate run
Check status	`systemctl status <timer>.timer`	Verify active state
View logs	`journalctl -u <service>.service -b`	Diagnose failures
Rebuild initramfs	`sudo mkinitcpio -P` / `sudo dracut -f`	Sync kernel modules

11. When to seek external help

The timer is part of a third‑party package that you cannot modify.
You encounter a CVE‑related kernel module issue (see cve.mitre.org for details).
The service is critical for security (e.g., a firewall rule generator) and downtime is unacceptable.

In such cases, consult the upstream project’s issue tracker (e.g., github.com/systemd/systemd) or the distribution’s support forums.