Removing Stale SSH known_hosts Entries

Cleaning up after a host rebuild with ssh-keygen

When a host rebuilds, its SSH key changes

A fresh install or a system rebuild swaps the host’s RSA/ED25519 key pair.
If you still have the old key in ~/.ssh/known_hosts, the next SSH connection will trigger a “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” message.
That warning is a safety feature, but it also blocks automation and forces users to manually edit the file.

The quickest, safest way to clean up stale entries is to let OpenSSH do the work with ssh-keygen -R. The command removes every key that matches the supplied hostname or IP, regardless of whether the entry is hashed or plain.

[Read More]

Fixing a Boot Stuck at Loading initramfs

Adding an NVMe drive on Ubuntu 24.04

When a new NVMe sticks, the boot can stall at “Loading initramfs”

Adding a second NVMe drive to an existing Ubuntu 24.04 installation is a quick way to get more storage or carve out a dedicated data partition.
Most of the time the kernel pulls the NVMe driver on its own, but a handful of mis‑configurations can make the system hang in the initramfs shell.
Typical symptoms:

  • The screen shows “Loading initramfs …” and then stops.
  • The prompt switches to a root shell (root@hostname:/#).
  • Nothing else happens until you manually exit or reboot.

Below is a hands‑on walk‑through of the most common culprits and how to fix them. I’ll assume you have a working Ubuntu 24.04 install and a spare NVMe device that you just plugged in.

[Read More]

Launch a temporary root process with `systemd‑run` instead of `sudo`

Run a single command as root with systemd‑run

When you just need to fire up one command as root and then drop back, sudo is the go‑to.
But if your distro already runs systemd, systemd‑run can be a leaner, more isolated way to do it—especially for short‑lived tasks.

What systemd‑run actually does

systemd‑run spawns a transient unit (either a scope or a service) that inherits the caller’s privileges and then elevates them to root when you add --user or --scope with --user.
Unlike sudo, the unit vanishes as soon as the command finishes, and you can sprinkle a handful of systemd properties on it to tighten the environment.

[Read More]

Pulling One File from a Borg Backup

Restoring a file without unpacking the archive

Why pull a single file from a Borg archive?

When a configuration file or a database dump is corrupted, you often need only that one file, not the whole archive. Extracting the entire archive just to get a single file wastes bandwidth, CPU, and disk space—especially when the archive lives on a remote server or in a cloud bucket.

Because Borg keeps the archive compressed and deduplicated, the cost of pulling a single file is proportional to the file’s size and the number of blocks it occupies. Below is a quick walk‑through for retrieving a single file efficiently, both locally and remotely, and piping the data through other tools for immediate processing or secure transfer.

[Read More]

Fixing SSH Password Prompts After Adding a Key

Correcting PubkeyAuthentication and key permissions

Why SSH Still Asks for a Password After Adding a Key

Adding a public key to ~/.ssh/authorized_keys is the first step toward password‑less login, but the SSH daemon can still fall back to password authentication for a handful of reasons. The most common culprit is a mis‑configured PubkeyAuthentication setting in /etc/ssh/sshd_config or a permissions problem that causes the server to ignore the key file.


1. Verify the Key Is Accepted by the Server

# On the client
ssh -vvv user@host

The -vvv flag prints the entire authentication dance. Look for lines like:

[Read More]

Fixing “Host key verification failed” After Renaming a Jump Host Server

The “Host key verification failed” error after renaming a jump host

When you rename a jump host (or any SSH‑accessible machine) the client still keeps the old hostname in its ~/.ssh/known_hosts. On the next connection the client pulls the key presented by the renamed host and compares it with the stored key for the old name. The mismatch triggers the classic

Host key verification failed.

It’s a safety net against man‑in‑the‑middle attacks, but it can be a nuisance when you legitimately change the hostname. Below is a practical, step‑by‑step fix, a few automation tricks, and some security notes I’ve learned after 20 years on the grind.

[Read More]

Fixing systemd Timer Failures After Reboot

Handling an unset USER variable in service units

Why the $USER Variable Matters for Systemd Timers

Systemd timers are the go‑to replacement for cron on almost every distro. They’re declarative, mesh cleanly with the unit graph, and bring a bunch of goodies—persistent timers, calendar syntax, and dependency handling. The catch? If your timer’s service unit runs a script that relies on $USER, you’ll see a silent failure after a reboot. Systemd doesn’t set $USER for system‑wide units; it only populates it for user‑specific units that run in a logged‑in session.

[Read More]

Converting a Nested JSON List of Users into a Simple CSV with jq and awk

When a service spits out a JSON blob that contains a list of users, the data is usually nested and has more fields than you actually need. Turning that into a flat CSV is a common thing to do for automation scripts, reporting, or feeding into spreadsheets. The two most reliable command‑line tools for this job are jq (a JSON processor) and awk (a text filter). Together they give you a fast, secure, and scriptable pipeline.

[Read More]

Fixing /tmp Permission Errors After a Kernel Update

Restoring the sticky bit on /tmp

The sticky bit in a nutshell

The sticky bit (+t) on a directory means that only the file’s owner, the directory’s owner, or root can delete or rename files inside it. /tmp is traditionally set to 1777 (octal) – owner root, group root, permissions rwxrwxrwt. That lets any user create files, but stops them from messing with each other’s data.

What happened after the update

Some kernel releases changed the default mount options for tmpfs or how tmpfs‑based /tmp is handled. If /tmp is mounted as tmpfs without the sticky bit, the kernel quietly clears the +t flag. The directory stays world‑writable, but the protection that stops users from deleting each other’s files disappears. Applications that expect the sticky bit now refuse to write, because they see the directory as insecure.

[Read More]

Fixing a Failed /boot Mount in Emergency Mode

Recovering Ubuntu 24.04 LTS after a kernel update

Why a kernel update can drop you into emergency mode because /boot won’t mount

A fresh kernel can change the initramfs or the boot‑loader’s config, and if the loader can’t find the /boot partition the system drops into that classic emergency shell:

Emergency shell (root=rw) – press Ctrl‑D to continue

Below is a no‑fluff, hands‑on walk‑through that keeps the commands you need, the trade‑offs, and a few security reminders.


1. Inspect the emergency environment

First, make sure the culprit is really the /boot mount. In the emergency shell run:

[Read More]