Open-Source

The first time I noticed the problem, a Docker‑based microservice was silently timing out on every outbound request. The logs were full of “no servers could be reached” errors, but the host machine could resolve the same hostnames without issue. After a quick check of the container’s /etc/resolv.conf, I saw an odd line: search mydomain.com. The host’s /etc/resolv.conf had no such entry. That single missing search domain was the root of the DNS failure inside containers.

Why /etc/hosts suddenly stops being respected after a kernel upgrade

When a kernel upgrade lands, you expect only low‑level changes. In practice, the upgrade can ripple through the entire userland, especially when systemd‑resolved is involved. A common symptom is that hostname lookups that used to resolve via /etc/hosts now fall back to DNS or fail entirely. The culprit is often a subtle change in how systemd‑resolved loads its configuration or how the Name Service Switch (NSS) library consults /etc/hosts.

Why per‑interface search domains matter

If a box has more than one NIC—like a wired LAN that talks to a corporate DNS and a Wi‑Fi that talks to a public ISP—mixing the two can bite you. A query for server will be sent to the DNS server of the interface that sent the request, but the search list that the resolver appends is taken from the global configuration unless you tell it otherwise. That means you can end up asking your public DNS for an internal host or, even worse, leaking internal names to the Internet. Per‑interface search domains keep the resolver tidy and stop accidental leakage.

Rebuilding the initramfs is the most reliable way to escape an emergency shell after a kernel upgrade that leaves the system stuck in emergency mode.
The problem usually stems from a mismatch between the running kernel and the modules or hooks that the initramfs contains.
Below is a step‑by‑step guide that covers the common causes, the exact commands you’ll run, and the security‑aware trade‑offs you should keep in mind.

Why Emergency Mode Happens After a Kernel Update

When you install a new kernel on Ubuntu 24.04, GRUB points to the new image, but the initramfs that ships with it must contain:

Understanding the 2775 Permission Set

The octal mode 2775 is the go‑to for giving a directory shared write access while keeping new files in the same group.

• 2 – set‑gid bit: new files inherit the directory’s group.
• 7 – owner gets rwx.
• 7 – group gets rwx.
• 5 – others get r-x.

On a path like /srv/shared it looks tidy: any member of the group can drop files, and those files stay in the same group for later collaboration. The flip side? Group write is a double‑edged sword. If the group contains people you don’t fully trust, or if the directory is exposed to a wider audience, the set‑gid bit can become a vector for privilege escalation or accidental data exposure.

Why a service can stay “activating” after a kernel upgrade

When you bump the kernel, systemd does a hard reset of almost everything.
If a unit you depend on never leaves the activating state, the rest of the boot can stall or the service simply never runs.
The usual suspect is a dependency on a kernel‑provided resource that is missing or delayed after the upgrade.

Below is a practical checklist that shows how to diagnose the issue, why it happens, and how to fix it without compromising security or stability.

Why systemd‑resolved matters

On most modern distros the DNS stack is no longer a simple /etc/resolv.conf.
systemd‑resolved runs in the background, pulls DNS servers from every interface, does caching, validates with DNSSEC, and even acts as a local DNS‑over‑TLS (DoT) proxy.
resolvectl (now systemd-resolve in newer releases) is the CLI that lets you look inside this stack and tweak it without hunting through config files.

If you run a home lab, self‑hosted services, or just want a snappier, more reliable lookup, mastering resolvectl can shave milliseconds off each query and give you a clear view of what the resolver is actually doing.