Open-Source

When a new NVMe sticks, the boot can stall at “Loading initramfs”

Adding a second NVMe drive to an existing Ubuntu 24.04 installation is a quick way to get more storage or carve out a dedicated data partition.
Most of the time the kernel pulls the NVMe driver on its own, but a handful of mis‑configurations can make the system hang in the initramfs shell.
Typical symptoms:

• The screen shows “Loading initramfs …” and then stops.
• The prompt switches to a root shell (root@hostname:/#).
• Nothing else happens until you manually exit or reboot.

Below is a hands‑on walk‑through of the most common culprits and how to fix them. I’ll assume you have a working Ubuntu 24.04 install and a spare NVMe device that you just plugged in.

When you fire up a heavy build, a data‑scraping job, or a local web server inside a tmux session, the usual pattern is to detach and log out. tmux will happily keep the session alive as long as your login shell stays around, but as soon as you hit logout or PAM times you out, the whole tree dies. Wrapping tmux in a systemd unit makes it immune to those surprises and gives you the same tooling you already use for other services—logging, restart policies, limits, and isolation.

Why JSON‑friendly syslog matters

Syslog is the default source for almost every Linux service. When an error happens, the kernel or a daemon writes a line that looks like this:

Oct  6 12:34:56 myhost kernel: [12345.678901] EXT4-fs error (device sda1): ext4_find_entry:1073: inode #123456: comm: myapp: file name too long

Grafana loves structured data. A plain text file is hard to query, filter, or aggregate. By converting the error lines to JSON you can:

The “Host key verification failed” error after renaming a jump host

When you rename a jump host (or any SSH‑accessible machine) the client still keeps the old hostname in its ~/.ssh/known_hosts. On the next connection the client pulls the key presented by the renamed host and compares it with the stored key for the old name. The mismatch triggers the classic

Host key verification failed.

It’s a safety net against man‑in‑the‑middle attacks, but it can be a nuisance when you legitimately change the hostname. Below is a practical, step‑by‑step fix, a few automation tricks, and some security notes I’ve learned after 20 years on the grind.

Why the $USER Variable Matters for Systemd Timers

Systemd timers are the go‑to replacement for cron on almost every distro. They’re declarative, mesh cleanly with the unit graph, and bring a bunch of goodies—persistent timers, calendar syntax, and dependency handling. The catch? If your timer’s service unit runs a script that relies on $USER, you’ll see a silent failure after a reboot. Systemd doesn’t set $USER for system‑wide units; it only populates it for user‑specific units that run in a logged‑in session.

When a service spits out a JSON blob that contains a list of users, the data is usually nested and has more fields than you actually need. Turning that into a flat CSV is a common thing to do for automation scripts, reporting, or feeding into spreadsheets. The two most reliable command‑line tools for this job are jq (a JSON processor) and awk (a text filter). Together they give you a fast, secure, and scriptable pipeline.

Why Move /var/log to a tmpfs?

On most production boxes and home‑lab rigs the root file system is a single ext4 or btrfs partition that also holds /var/log. After a few months the logs can eat up gigabytes of space, especially on servers that churn out audit, kernel, or app logs. A full root partition can block package upgrades, kernel updates, or even prevent a boot if the filesystem gets remounted read‑only. Moving /var/log to a tmpfs frees the disk right away, keeps the logs in RAM, and cuts out the need to compress or rotate them manually.

Why a kernel update can drop you into emergency mode because /boot won’t mount

A fresh kernel can change the initramfs or the boot‑loader’s config, and if the loader can’t find the /boot partition the system drops into that classic emergency shell:

Emergency shell (root=rw) – press Ctrl‑D to continue

Below is a no‑fluff, hands‑on walk‑through that keeps the commands you need, the trade‑offs, and a few security reminders.

1. Inspect the emergency environment

First, make sure the culprit is really the /boot mount. In the emergency shell run:

Why Bluetooth headphones keep dropping on GNOME 48

Bluetooth audio on GNOME 48 is now handled by PipeWire instead of the old PulseAudio stack. The new stack is cleaner, but it also turns on a “low‑power” mode that can make a headset drop after a few minutes of idle time or when you close the laptop lid. The real culprit is usually the kernel’s power‑management for the Bluetooth controller and PipeWire’s default settings.