A 20 GB VPS is a tight fit. On a busy host, the system journal can grow fast, especially with frequent reboots, kernel messages, or misconfigured services that spam systemd-journald. When the journal consumes several gigabytes, you hit the “no space left on device” error on /var/log or /var/run, and the OS starts throttling services or, worse, refuses to start new ones.
systemd‑resolved is giving you 127.0.0.1 for every hostUbuntu 24.04 ships with systemd‑resolved as the DNS resolver.
If every hostname you look up ends up at 127.0.0.1, the most common culprit is the search‑domain order that resolved builds.
It pulls the list from three places:
/etc/resolv.conf (or the DNS=/Domains= directives in /etc/systemd/resolved.conf)$HOSTNAME)When localhost or the local hostname lands at the top, the resolver will try it before forwarding the query to the real DNS servers, and you’ll get a loopback address for any name that isn’t fully qualified.
When a server’s RAM jumps from 30 % to 90 % in a matter of minutes, the first instinct is to kill processes, restart services, or add more memory. Those are quick fixes, but they don’t tell you why the spike happened. On a system that runs systemd, the combination of journalctl and systemd‑coredump gives a precise, timestamp‑aligned view of what went wrong. This post walks through a real‑world scenario, shows how to extract the relevant logs, and explains how to interpret the data so you can fix the root cause and harden the system against future incidents.
You tweak a unit file, hit systemctl, and nothing happens.
That’s the classic “I changed the file, but systemd still runs the old version” problem.
The daemon is still holding the old config in memory. systemd keeps a cache of unit files; it only rereads them after a daemon‑reload. If you skip that step, the changes stay invisible until you reboot or reload manually.
The sticky bit (+t) on a directory means that only the file’s owner, the directory’s owner, or root can delete or rename files inside it. /tmp is traditionally set to 1777 (octal) – owner root, group root, permissions rwxrwxrwt. That lets any user create files, but stops them from messing with each other’s data.
Some kernel releases changed the default mount options for tmpfs or how tmpfs‑based /tmp is handled. If /tmp is mounted as tmpfs without the sticky bit, the kernel quietly clears the +t flag. The directory stays world‑writable, but the protection that stops users from deleting each other’s files disappears. Applications that expect the sticky bit now refuse to write, because they see the directory as insecure.
When a new kernel lands, the kernel‑specific modules and initramfs changes can
break the ExecStartPre or ExecStart scripts that a timer relies on.
The result? A timer that never fires, or that fires with the wrong environment.
Below is a step‑by‑step guide to diagnose and restore a broken timer, with security‑aware notes and trade‑offs you’ll run into in a production or homelab setup.
[Read More]Kernel upgrades are usually painless, but on a few systems they can trigger a cascade of “carrier lost” messages that make the interface drop and come back up repeatedly. The messages are buried deep in the system journal, so you may not notice them until the network flaps start causing outages. This post shows how to locate those hidden messages with journalctl, correlate them with real‑world symptoms, and apply a fix that keeps the interface stable.
/etc/hosts, the file often ends up with Windows‑style CRLF line endings.If you’ve ever pulled a hosts file from a Windows share, you’ve probably noticed that the resolver starts acting weird. The glibc resolver stops at the carriage return (\r) and treats the rest of the line as part of the hostname. That means lookups for localhost, myserver, or any entry that follows a stray CR fail, and some services silently fall back to DNS—exposing the system to name‑resolution attacks.
I’ve seen this go wrong when a service is misconfigured and ends up consuming all your system resources. Systemd is a core component of most modern Linux distributions, responsible for managing system services. One of its key features is the ability to automatically restart services that fail or exit unexpectedly. However, if not properly configured, this can lead to a “service deluge” where a failing service is repeatedly restarted.
[Read More]When your Linux system fails to boot, it can be frustrating - especially if you’re not familiar with the troubleshooting process. I’ve seen this go wrong when people don’t know where to start. Luckily, systemd’s Emergency Mode and Rescue Shell are here to help. In this article, we’ll dive into how to use these tools to troubleshoot initramfs issues and get your system up and running again.
[Read More]