Fixing systemd-resolved Ignoring /etc/hosts

Resolving local hostname lookups that fail silently

Ever had a host you added to /etc/hosts just stop answering? I’ve seen this happen after a kernel upgrade or a quick edit that left a stray comment. On Fedora, Debian, and Arch the culprit is almost always the local resolver: systemd‑resolved. It only reads /etc/hosts if the ResolveFile option is enabled and the service is started with --resolve. A tiny mis‑config can silence that hook.


1. Check that systemd‑resolved is actually running

systemctl status systemd-resolved.service

If you see inactive or failed, start it up:

[Read More]

Quickly Pinpointing Unauthorised sudo Attempts with Journalctl Filters

When a user or a process tries to run something with sudo, the action is recorded in the system journal. Because sudo writes to the journal, you can filter on the SYSLOG_IDENTIFIER or MESSAGE fields to spot failed attempts, audit who tried what, and even trigger alerts.

Below is a step‑by‑step guide that shows how to:

  • Identify failed sudo attempts in real time
  • Pull historical data for a specific user or time window
  • Distinguish between legitimate and suspicious activity
  • Automate the process with a small script
  • Understand trade‑offs and common pitfalls

The examples assume a recent systemd‑based distro (Debian 12, Ubuntu 24.04, Fedora 40, etc.) and that sudo is compiled with the default --syslog flag (enabled since 2019). If your system uses a custom sudo build or a different logging backend, the commands will need adjustment.

[Read More]

Fixing systemd-resolved Dropping /etc/hosts Entries

When a local service becomes unreachable

systemd‑resolved is the default DNS resolver on most modern distros. It pulls /etc/hosts unless you flip the --no-hosts flag. In 2026 a subtle tweak in its default behaviour started swallowing local hostname lookups, and no one noticed until a service went down.

Why it happens

systemd-resolved now gives the DNS stack a higher priority than the hosts file when you call getaddrinfo(3). If a name shows up in both /etc/hosts and a DNS zone, the DNS answer wins. When the zone is empty or the query times out, it silently falls back to the hosts file. But if the DNS server is unreachable and the hosts entry points to 127.0.0.1 or ::1, the resolver drops that entry and returns ENOTFOUND. The change was meant to stop a compromised DNS server from poisoning local lookups.

[Read More]

Clearing Stale DNS Entries from resolvectl

Fixing resolver caches without restarting NetworkManager

When a host suddenly resolves a domain to an old IP, the first instinct is to restart NetworkManager or reboot the machine. That works, but it’s heavy‑handed and can disrupt services. In a production environment, a brief outage can trigger SLA violations or expose stale data to clients. Below is a concise, hands‑on guide to flushing the systemd‑resolved cache safely, with a focus on security‑aware operators who need minimal downtime.

[Read More]

Why Nginx Stops After a Kernel Upgrade

Debugging a systemd timer on a home-lab server

I upgraded the kernel on my 2026‑era Debian‑based home‑lab server from 6.1.12‑generic to 6.5.0‑generic. The installer was a breeze – “All packages up to date” and a clean reboot. A few hours later, Nginx stopped answering HTTP. The systemctl status nginx looked like this:

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2026‑11‑06 14:32:12 UTC; 1h 12min ago
  Process: 1234 ExecStart=/usr/sbin/nginx -g daemon off; (code=exited, status=0/SUCCESS)
   Main PID: 1234 (code=exited, status=0/SUCCESS)

The master exited cleanly, yet the port was closed. The only clue? The kernel had changed. Below is the debugging walk‑through, the root cause, and how to harden the system so a kernel upgrade never silently kills a critical service again.

[Read More]

Turning a Splunk Dashboard into a Threat-Hunting Playbook

Building practical detection workflows from existing data

Turning a Splunk Dashboard into a Hands‑On Threat‑Hunting Playbook

Splunk dashboards are great for visualizing data, but they rarely give you a ready‑to‑execute hunting workflow. In a real‑world environment you need a playbook that tells you what to look for, how to filter the data, and what to do when you find something. This post walks through turning an existing Splunk dashboard into a fully documented playbook that can be run by a sysadmin or a SOC analyst on a Linux host. The focus is on practical commands, trade‑offs, and troubleshooting tips that fit the Linux‑security‑first mindset of the Linux Security blog.

[Read More]

Disabling Unnecessary systemd Units

A one-line command that keeps logs clean

Why disabling unused systemd units matters

Every fresh install of a modern distro ships dozens of systemd units.
Bluetooth, CUPS, smart‑card readers, legacy daemons—you name it.
On a headless server most of those are overkill. They still eat a bit of RAM,
they create audit events that you’ll never look at, and they’re an extra attack surface.

The quickest way to shrink that surface and keep the journal from turning into a log‑dump is to disable the unit in one go and clean up its log space.

[Read More]

Freeing up 5 GB of disk space by pruning old journalctl logs on a 20 GB VPS

Freeing 5 GB of Disk Space by Pruning Old journalctl Logs on a 20 GB VPS

A 20 GB VPS is a tight fit. On a busy host, the system journal can grow fast, especially with frequent reboots, kernel messages, or misconfigured services that spam systemd-journald. When the journal consumes several gigabytes, you hit the “no space left on device” error on /var/log or /var/run, and the OS starts throttling services or, worse, refuses to start new ones.

[Read More]

Fixing systemd-resolved Returning 127.0.0.1

Correcting search domain order on Ubuntu 24.04

Why systemd‑resolved is giving you 127.0.0.1 for every host

Ubuntu 24.04 ships with systemd‑resolved as the DNS resolver.
If every hostname you look up ends up at 127.0.0.1, the most common culprit is the search‑domain order that resolved builds.
It pulls the list from three places:

  1. /etc/resolv.conf (or the DNS=/Domains= directives in /etc/systemd/resolved.conf)
  2. DHCP‑provided domains
  3. The machine’s own hostname ($HOSTNAME)

When localhost or the local hostname lands at the top, the resolver will try it before forwarding the query to the real DNS servers, and you’ll get a loopback address for any name that isn’t fully qualified.

[Read More]

Tracing a Sudden Memory Spike with journalctl

Using systemd-coredump logs to find the cause

Sudden Memory Spike? Use journalctl + systemd‑coredump to Pin It Down

When a server’s RAM jumps from 30 % to 90 % in a matter of minutes, the first instinct is to kill processes, restart services, or add more memory. Those are quick fixes, but they don’t tell you why the spike happened. On a system that runs systemd, the combination of journalctl and systemd‑coredump gives a precise, timestamp‑aligned view of what went wrong. This post walks through a real‑world scenario, shows how to extract the relevant logs, and explains how to interpret the data so you can fix the root cause and harden the system against future incidents.

[Read More]