On a busy host, a 1‑second DNS lookup can feel like a blink, but when it happens on every request, it adds up. In 2026, most distributions ship systemd‑resolved as the default resolver, and the default configuration is usually fine. Yet a handful of real‑world scenarios—high‑traffic web servers, containers, or networks with strict firewall rules—expose a subtle 1‑second delay that can be traced back to systemd‑resolved’s handling of upstream DNS servers.
[Read More]Recovering a Kernel Panic After Adding a New Initramfs Module
Kernel Panic After Adding a New Initramfs Module – How to Recover
Dropping a new module into an initramfs and watching the machine go black is a classic “what‑did‑I‑break‑this” moment. In the real world, the culprit is almost never the module itself; it’s usually a mis‑configured initramfs, a missing dependency, or a kernel‑level feature that got turned off by accident. Below is a practical, step‑by‑step recovery guide that keeps hardening in mind, gives you concrete commands, and explains the trade‑offs you’ll face when rebuilding the initramfs.
[Read More]Multi‑jump bastion access without storing keys: practical ssh‑agent forwarding
Multi‑jump Bastion Access Without Storing Keys
When you need to reach a private host behind a bastion, the first thing most people do is drop a private key on the jump machine and use it to hop over. That’s quick, but it gives the bastion a copy of the key it never should have. A cleaner pattern is to forward your local ssh‑agent through the bastion so the key never lands on the jump host.
[Read More]Using SSH Agent Forwarding Safely
Avoiding key leaks when jumping through bastions
When you have to touch a host that lives behind several bastions, forwarding the agent is the quickest way to keep your private key off every hop. The catch? A compromised bastion can steal that key if you’re not careful. Below is a hands‑on walk‑through that shows how to tighten forwarding, what to watch out for, and how to double‑check that the key never ends up where it shouldn’t.
1. Understand the Threat Model
| Threat | Why it matters |
|---|---|
| Key exposure | A breached bastion can use the forwarded key to reach downstream hosts. |
| MITM | An attacker could trick your local SSH client into forwarding the key to an arbitrary host. |
| Untrusted hosts | Some internal services (monitoring agents, etc.) might run an SSH server that could request the key. |
The aim is to confine the key to the exact hosts you trust and to stop accidental leaks.
[Read More]Fixing SSH Jump Host Drops After OpenSSH 9.5
Diagnosing silent connection failures through a bastion
SSH jump host silently dropping connections after OpenSSH 9.5
A recent OpenSSH 9.5 release tweaked how the server handles TCP keepalive packets. My jump host – a modest Ubuntu 24.04 box – started dropping established SSH sessions after a few minutes of idle time. The problem popped up during a routine audit when several users complained of “connection closed” messages, yet nothing appeared in the logs.
What changed in 9.5?
OpenSSH 9.5 introduced new defaults for ClientAliveInterval and ClientAliveCountMax on the server side:
Why Nginx Stops After a Kernel Upgrade
Debugging a systemd timer on a home-lab server
I upgraded the kernel on my 2026‑era Debian‑based home‑lab server from 6.1.12‑generic to 6.5.0‑generic. The installer was a breeze – “All packages up to date” and a clean reboot. A few hours later, Nginx stopped answering HTTP. The systemctl status nginx looked like this:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2026‑11‑06 14:32:12 UTC; 1h 12min ago
Process: 1234 ExecStart=/usr/sbin/nginx -g daemon off; (code=exited, status=0/SUCCESS)
Main PID: 1234 (code=exited, status=0/SUCCESS)
The master exited cleanly, yet the port was closed. The only clue? The kernel had changed. Below is the debugging walk‑through, the root cause, and how to harden the system so a kernel upgrade never silently kills a critical service again.
[Read More]Hardening Serverless Function IAM Roles
Fixing over-permissive execution permissions
When a serverless function is granted a blanket AdministratorAccess policy, the blast radius of a compromised function is the entire account. In 2025, a handful of high‑profile breaches were traced back to overly permissive IAM roles attached to Lambda, Cloud Functions, or Azure Functions. The fix is simple: reduce the scope of the role to the minimum set of actions required by the function. Below is a practical walkthrough that blends AWS, Azure, and GCP examples, with a focus on Linux sysadmins who are comfortable with CLI, JSON, and policy evaluation.
The Cost of Over-Broad Backup Retention
How a 30-day policy exposed 2 TB of PHI
A 30‑day retention window is a default in many backup scripts and commercial tools. It feels safe: “I’ll keep the last month of data, that’s enough for recovery.” In 2025, a mid‑size clinic used exactly that policy on a set of encrypted LVM snapshots stored on a single NAS. When the NAS firmware update rolled out, the snapshots were corrupted, and the backup system fell back to a restore‑all mode. The result? 2 TB of unencrypted PHI leaked to a rogue process that had gained write access to the backup directory.
[Read More]Disabling Unnecessary systemd Units
A one-line command that keeps logs clean
Why disabling unused systemd units matters
Every fresh install of a modern distro ships dozens of systemd units.
Bluetooth, CUPS, smart‑card readers, legacy daemons—you name it.
On a headless server most of those are overkill. They still eat a bit of RAM,
they create audit events that you’ll never look at, and they’re an extra attack
surface.
The quickest way to shrink that surface and keep the journal from turning into a log‑dump is to disable the unit in one go and clean up its log space.
[Read More]Fixing SSH Password Prompts After Adding a Key
Correcting PubkeyAuthentication and key permissions
Why SSH Still Asks for a Password After Adding a Key
Adding a public key to ~/.ssh/authorized_keys is the first step toward password‑less login, but the SSH daemon can still fall back to password authentication for a handful of reasons. The most common culprit is a mis‑configured PubkeyAuthentication setting in /etc/ssh/sshd_config or a permissions problem that causes the server to ignore the key file.
1. Verify the Key Is Accepted by the Server
# On the client
ssh -vvv user@host
The -vvv flag prints the entire authentication dance. Look for lines like: