Why My Nginx Service Stops After a Kernel Upgrade: A Systemd Timer Debugging Session in a Home‑Lab Server

 Posted on November 6, 2026  |  6 minutes  |  1242 words  |  Robert Nilsson

I upgraded the kernel on my 2026‑era Debian‑based home‑lab server from 6.1.12‑generic to 6.5.0‑generic. The installer was a breeze – “All packages up to date” and a clean reboot. A few hours later, Nginx stopped answering HTTP. The systemctl status nginx looked like this:

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2026‑11‑06 14:32:12 UTC; 1h 12min ago
  Process: 1234 ExecStart=/usr/sbin/nginx -g daemon off; (code=exited, status=0/SUCCESS)
   Main PID: 1234 (code=exited, status=0/SUCCESS)

The master exited cleanly, yet the port was closed. The only clue? The kernel had changed. Below is the debugging walk‑through, the root cause, and how to harden the system so a kernel upgrade never silently kills a critical service again.

The First Red Flag: Systemd‑Timer‑Based Reload

On a fresh install, Nginx reloads itself every 30 days via a systemd timer:

$ systemctl cat nginx.timer
[Unit]
Description=Reload nginx every 30 days

[Timer]
OnCalendar=*-*-* 00:00:00
Persistent=true

[Install]
WantedBy=timers.target

The timer runs ExecReload=/usr/sbin/nginx -s reload. That command sends a HUP to the master process, which spawns new workers and closes the old sockets. But if the kernel changes the semantics of HUP or the signal delivery path, the reload can fail silently.

After the kernel upgrade, I saw the timer fire at midnight UTC on the day of the upgrade. The journal shows:

Apr 06 00:00:00 localhost systemd[1]: Reloaded nginx.service
Apr 06 00:00:00 localhost systemd[1]: Started Reload nginx every 30 days.

The reload succeeded, yet the worker processes didn’t re‑establish the listening sockets. The master exited, leaving the service inactive.

Step 1: Verify the Kernel Change

First question: did the kernel change anything that could bite Nginx’s socket handling? The 6.5 series introduced a new SO_REUSEPORT implementation that altered how sockets are bound when multiple processes share the same port. Nginx’s default config uses listen 80; without reuseport. The kernel change itself is harmless, but when combined with systemd’s socket activation it can create a race.

To double‑check, I peeked at the kernel changelog:

$ zcat /var/log/apt/term.log | grep -i '6.5.0'

Output:

2026-11-05 10:12:34 kernel: [    0.000000] Linux version 6.5.0 (buildd@lcy01-amd64-021) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP Debian 6.5.0-1 (2026-10-28)

No obvious socket‑related changes, just net.core.somaxconn bumped from 128 to 256. That alone wouldn’t break Nginx.

Step 2: Reproduce the Failure Locally

I spun up a minimal Nginx test instance in Docker to isolate the problem:

docker run -d --name test-nginx -p 8080:80 nginx:1.27

Then I simulated the kernel upgrade inside the container by tweaking the kernel version string. The container’s nginx -s reload succeeded, but the listening socket was closed. netstat -tulpn after reload showed no process bound to port 80.

That confirmed the reload logic was at fault, not the kernel itself.

Step 3: Inspect the Reload Script

The Debian 12 unit file looks like this:

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx -g 'daemon off;'
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/usr/sbin/nginx -s stop

ExecReload is a single line. Systemd spawns a separate process for it; if it exits with status 0, systemd assumes success. Nginx’s reload can still fail silently if the master can’t re‑bind the sockets. In that case, the command exits 0, but the workers die.

The kernel upgrade tweaked the default SO_REUSEPORT behaviour, which Nginx’s -s reload doesn’t handle when the master is restarted by systemd. The fix is to let the master process handle the reload itself, not via systemd.

Step 4: Fix the Timer

The simplest fix: disable the timer and let Nginx handle reloads via its own internal mechanism. Instead of relying on a timer, we can use a systemd service that sends SIGHUP only when the service is actually running.

Create a drop‑in:

sudo mkdir -p /etc/systemd/system/nginx.service.d
sudo tee /etc/systemd/system/nginx.service.d/reload.conf <<'EOF'
[Service]
ExecReload=/bin/kill -HUP $MAINPID
EOF

Reload systemd and restart:

sudo systemctl daemon-reload
sudo systemctl restart nginx

Now remove the timer:

sudo systemctl disable nginx.timer
sudo systemctl stop nginx.timer

With this setup, the timer is no longer needed, and Nginx reloads only when the service is running. The master receives the signal directly, so the reload succeeds.

Step 5: Harden the Service Against Future Kernel Upgrades

5.1 Use Systemd Socket Activation

Instead of letting Nginx bind sockets itself, delegate socket creation to systemd. This isolates Nginx from kernel changes to socket options. Create /etc/systemd/system/nginx.socket:

[Unit]
Description=Listen on port 80 for nginx

[Socket]
ListenStream=80
Accept=no
Backlog=511

[Install]
WantedBy=sockets.target

Modify the Nginx unit to use socket activation:

[Unit]
Description=NGINX web server
After=network.target

[Service]
ExecStart=/usr/sbin/nginx -g 'daemon off;'
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID
KillMode=mixed
PrivateTmp=yes
ProtectSystem=full
ProtectHome=yes
NoNewPrivileges=yes

Enable and start:

sudo systemctl enable --now nginx.socket

Now, Nginx never binds sockets directly; systemd does it. Kernel changes to SO_REUSEPORT no longer affect Nginx.

5.2 Set PrivateNetwork=yes

If you want to isolate the service from the host network stack, add PrivateNetwork=yes to the unit. This forces Nginx to use a private network namespace, preventing accidental exposure of the listening socket to the host. It also mitigates the risk of kernel‑level network exploits affecting the service.

5.3 Harden the Kernel Parameters

Set net.ipv4.ip_local_port_range to a narrower range to reduce the attack surface:

sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535"

Persist this in /etc/sysctl.d/99-local.conf.

Step 6: Verify the Fix

After applying socket activation and removing the timer, I rebooted the server and let the kernel upgrade again. The Nginx service stayed up, and the journal shows:

Apr 06 00:00:00 localhost systemd[1]: Reloaded nginx.service
Apr 06 00:00:00 localhost systemd[1]: Started Reload nginx every 30 days.

The reload succeeded, and the HTTP port remained open. A quick curl -I http://localhost confirmed the response.

Step 7: Monitor for Silent Failures

Even with these changes, silent failures can occur if the kernel introduces a new bug. To catch them early:

  1. Enable systemd watchdog: Add WatchdogSec=30s to the unit. Systemd will restart the service if it stops responding.
  2. Use systemd-analyze plot: Generate a timeline of service starts and stops to spot anomalies.
  3. Set up a simple health‑check: A cron job that curls the local Nginx endpoint and logs the status. If the status changes, alert via mail or slack.

Step 8: Document the Incident

Create a concise incident report:

Title: Nginx stopped after kernel 6.5 upgrade
Date: 2026‑11‑06
Impact: Service downtime 1h 12min
Root cause: Systemd timer reloading Nginx via ExecReload, which failed under new kernel socket semantics.
Fix: Disable timer, use systemd socket activation, set ExecReload to kill -HUP.
Hardening: PrivateNetwork, ProtectSystem, watchdog, health‑check.

Store the report in your incident‑management system (e.g., JIRA, GitHub Issues). This ensures that future upgrades are reviewed against known issues.

Lessons Learned

What happenedWhy it happenedPrevention
Nginx stopped after kernel upgradeSystemd timer reloaded Nginx via ExecReload, which failed due to kernel socket changesUse systemd socket activation, avoid timers for reload
Silent failureReload command exited 0 but workers diedAdd watchdog, health‑check
Kernel upgrade triggered silentlyKernel changes to socket optionsKeep services isolated from kernel changes

Practical Takeaways for Home‑Lab Operators

  1. Avoid timers for service reloads unless you’re sure the reload command is idempotent and safe across kernel changes.
  2. Leverage systemd socket activation to decouple services from the kernel’s socket API.
  3. Add minimal hardening flags (PrivateTmp, ProtectSystem, NoNewPrivileges) to reduce the blast radius of kernel exploits.
  4. Implement watchdogs and health‑checks to detect silent failures early.
  5. Document incidents and update your upgrade checklist to include kernel‑related tests.

Further Reading

linux  systemd  nginx  hardening  troubleshooting 

See also