Systemd on Linux Security

Launch a temporary root process with `systemd‑run` instead of `sudo`

Fri, 09 Oct 2026 11:54:11 +0200

Run a single command as root with `systemd‑run`

When you just need to fire up one command as root and then drop back, sudo is the go‑to.
But if your distro already runs systemd, systemd‑run can be a leaner, more isolated way to do it—especially for short‑lived tasks.

What `systemd‑run` actually does

systemd‑run spawns a transient unit (either a scope or a service) that inherits the caller’s privileges and then elevates them to root when you add --user or --scope with --user.
Unlike sudo, the unit vanishes as soon as the command finishes, and you can sprinkle a handful of systemd properties on it to tighten the environment.

Fixing “Permission denied” on /tmp After a Kernel Update Removed the Sticky Bit

Wed, 30 Sep 2026 11:22:30 +0200

The sticky bit in a nutshell

The sticky bit (+t) on a directory means that only the file’s owner, the directory’s owner, or root can delete or rename files inside it. /tmp is traditionally set to 1777 (octal) – owner root, group root, permissions rwxrwxrwt. That lets any user create files, but stops them from messing with each other’s data.

What happened after the update

Some kernel releases changed the default mount options for tmpfs or how tmpfs‑based /tmp is handled. If /tmp is mounted as tmpfs without the sticky bit, the kernel quietly clears the +t flag. The directory stays world‑writable, but the protection that stops users from deleting each other’s files disappears. Applications that expect the sticky bit now refuse to write, because they see the directory as insecure.

How to Get a Systemd Timer Back on Track After a Kernel Upgrade

Fri, 18 Sep 2026 09:33:58 +0200

Systemd timers can fall silent after a kernel upgrade

When a new kernel lands, the kernel‑specific modules and initramfs changes can break the ExecStartPre or ExecStart scripts that a timer relies on.
The result? A timer that never fires, or that fires with the wrong environment.

Below is a step‑by‑step guide to diagnose and restore a broken timer, with security‑aware notes and trade‑offs you’ll run into in a production or homelab setup.

Using journalctl to Track Down the Hidden ‘eth0’ Carrier Lost Messages That Cause Network Flaps After a Kernel Upgrade

Mon, 14 Sep 2026 11:45:14 +0200

When a kernel upgrade silently breaks your network

Kernel upgrades are usually painless, but on a few systems they can trigger a cascade of “carrier lost” messages that make the interface drop and come back up repeatedly. The messages are buried deep in the system journal, so you may not notice them until the network flaps start causing outages. This post shows how to locate those hidden messages with journalctl, correlate them with real‑world symptoms, and apply a fix that keeps the interface stable.

Taming systemd Restart Policies to Prevent Service Deluge

Sat, 15 Aug 2026 08:22:06 +0200

Introduction to systemd Restart Policies

I’ve seen this go wrong when a service is misconfigured and ends up consuming all your system resources. Systemd is a core component of most modern Linux distributions, responsible for managing system services. One of its key features is the ability to automatically restart services that fail or exit unexpectedly. However, if not properly configured, this can lead to a “service deluge” where a failing service is repeatedly restarted.

Recovering from a Failed Boot: Using systemd's Emergency Mode and Rescue Shell to Troubleshoot Initramfs Issues

Thu, 13 Aug 2026 08:04:43 +0200

Introduction to Emergency Mode and Rescue Shell

When your Linux system fails to boot, it can be frustrating - especially if you’re not familiar with the troubleshooting process. I’ve seen this go wrong when people don’t know where to start. Luckily, systemd’s Emergency Mode and Rescue Shell are here to help. In this article, we’ll dive into how to use these tools to troubleshoot initramfs issues and get your system up and running again.

Taming Log Noise with journalctl: Filtering Out the Unimportant Stuff

Wed, 12 Aug 2026 10:40:19 +0200

Introduction to Log Noise

When working with Linux systems, logs are an essential part of troubleshooting, monitoring, and maintaining the health of your system. However, with the vast amount of data being logged, it can become overwhelming to sift through the noise to find the important information. I’ve seen this go wrong when trying to debug a complex issue, only to get lost in a sea of irrelevant log messages. This is where journalctl comes in, a powerful tool for managing and filtering system logs.

Troubleshooting systemd Service Restart Failures with Dependency Ordering and Retry Policies

Tue, 11 Aug 2026 10:39:09 +0200

Introduction to systemd Service Restart Failures

When working with Linux systems, especially those using systemd as the init system, service management is crucial for maintaining system stability and functionality. I’ve seen this go wrong when a service fails to restart properly, often due to dependency ordering issues or retry policy misconfigurations. Understanding how to troubleshoot and resolve these issues is essential for ensuring system reliability and uptime.

Understanding systemd Service Dependencies

Systemd services can have complex dependencies, defined in their service files (typically located in /etc/systemd/system/ or /usr/lib/systemd/system/). These dependencies are crucial for ensuring that services start in the correct order. For example, a web server might depend on the network service to be started before it can operate. Misconfigured dependencies can lead to services failing to start or restart. The real trick is to understand the startup sequence and identify potential bottlenecks.

Troubleshooting systemd Service Restart Loops with Dependency Ordering

Sat, 08 Aug 2026 11:01:52 +0200

Introduction to systemd Service Restart Loops

I’ve seen this go wrong when working with Linux systems: a service enters a restart loop, and it’s not immediately clear why. These loops occur when a service fails to start or restarts repeatedly, often due to dependency issues or configuration problems. In practice, identifying the root cause can be tricky, but there are some practical steps you can take to troubleshoot systemd service restart loops.

Using systemd to Manage and Rotate Log Files Without Running Out of Disk Space

Thu, 06 Aug 2026 10:21:43 +0200

Introduction to Log Rotation with systemd

I’ve seen log files consume all available disk space, bringing systems to a grinding halt. That’s why log rotation is a crucial aspect of system administration. With systemd becoming a de facto standard for managing system services and logs on many Linux distributions, I’ll explore how to use it to manage and rotate log files efficiently.

Understanding systemd-journald

systemd-journald is the system service that collects and stores log messages from various sources, including system services, kernel messages, and user applications. By default, journald stores log messages in a binary format, which can be queried using the journalctl command. To manage log rotation, we need to understand how journald handles log storage and rotation. Don’t bother with manually rotating log files - journald can handle it for you.

Taming systemd-resolved: Avoiding DNS Surprises on Multi-Network Linux Setups

Wed, 05 Aug 2026 08:04:38 +0200

Introduction to systemd-resolved

I’ve seen systemd-resolved cause some headaches in multi-network setups, so it’s worth understanding how it works. As part of the systemd suite, systemd-resolved is a DNS resolver that’s enabled by default on many modern Linux distributions, including Ubuntu and Fedora. The real trick is configuring it to avoid DNS surprises.

Configuring systemd-resolved

To get systemd-resolved working the way you want, you need to know how it uses DNS servers. It combines the DNS servers provided by your system’s network configuration with those specified in the /etc/systemd/resolved.conf file. Don’t bother with the network configuration; just modify the resolved.conf file to specify your preferred DNS servers. For example, to use Google’s public DNS servers, add the following lines to the [Resolve] section:

Taming Log Noise with jq and systemd Journal Filters

Tue, 04 Aug 2026 10:28:50 +0200

Introduction to Log Noise

I’ve seen log noise become a significant headache for Linux administrators, making it tough to identify important events and errors in system logs. With the sheer volume of data generated by modern systems, filtering out irrelevant information is crucial for efficient troubleshooting and maintenance. In this article, I’ll show you how to tame log noise using jq and systemd journal filters.

Understanding systemd Journal

The systemd journal is a centralized logging solution that collects and stores log messages from various system components. It’s a robust and efficient way to manage logs, allowing administrators to filter, prioritize, and analyze log data. The journal stores log messages in a binary format, which can be queried and filtered using the journalctl command. Don’t bother with trying to parse the binary format directly - it’s not worth the hassle.

Using resolvectl to Diagnose and Resolve Stubborn DNS Issues on Linux

Mon, 03 Aug 2026 11:32:32 +0200

Introduction to resolvectl

I’ve seen my fair share of DNS issues on Linux, and one tool that’s always helped me resolve them is resolvectl. This command-line utility, part of the systemd suite, provides a comprehensive way to query and configure DNS settings on your system. In this article, I’ll share some practical examples of how to use resolvectl to diagnose and fix stubborn DNS issues.

Understanding DNS Resolution

Before diving into resolvectl, it’s essential to understand how DNS resolution works on Linux. The DNS (Domain Name System) is responsible for translating human-readable domain names into IP addresses that computers can understand. On Linux systems, DNS resolution is typically handled by a resolver, which queries DNS servers to resolve domain names. The resolver configuration is usually defined in /etc/resolv.conf, but with systemd, resolvectl offers a more integrated and dynamic way to manage DNS settings.

Taming Systemd Service Restart Policies to Prevent Cascading Failures

Thu, 30 Jul 2026 11:07:07 +0200

Introduction to Systemd Service Restart Policies

Systemd is the backbone of most modern Linux distributions, and one of its key features is managing service restart policies. I’ve seen this go wrong when a service fails and takes down the entire system with it. In this article, we’ll dive into the world of systemd service restart policies and explore how to configure them to prevent cascading failures.

Understanding Systemd Service Restart Policies

Systemd provides several restart policies that can be applied to services. The real trick is understanding when to use each one:

Taming Log Noise with jq and systemd Journal Filters

Wed, 29 Jul 2026 11:23:22 +0200

Introduction to Log Noise

I’ve seen this go wrong when dealing with system logs - the sheer amount of unnecessary or redundant log messages can make it difficult to identify real issues. As a Linux administrator, managing log noise is essential for efficient system maintenance and troubleshooting. In this article, I’ll share how to tame log noise using jq and systemd journal filters.

Understanding systemd Journal

Systemd journal is a powerful centralized logging system that collects and stores log messages from various system components. It provides a robust and flexible way to manage logs, including filtering, prioritization, and storage. The real trick is to leverage the journalctl command to work with systemd journal.

Taming log Noise with journalctl: Filtering Out the Chaff to Find Meaningful Errors

Tue, 28 Jul 2026 08:12:37 +0200

Introduction to Journalctl

I’ve been working with Linux systems for years, and one tool that’s become essential to my workflow is journalctl. As of 2026, most modern Linux distributions have adopted systemd as their default init system, making journalctl a must-know tool for system administrators and users alike. In this article, I’ll share how to use journalctl to filter out unnecessary log noise and focus on meaningful errors.

Understanding Journalctl Basics

Before diving into advanced filtering techniques, let’s cover the basics of journalctl. The command journalctl can be used to view system logs, and by default, it will show you the most recent log entries. You can use the -n option to specify the number of log entries to display, for example, journalctl -n 10 will show you the last 10 log entries. Don’t bother with sudo unless you need to access system logs - journalctl will automatically use your current user’s permissions.

Debugging Connectivity Issues with resolvectl and ss: A Step-by-Step Guide to Identifying DNS and Routing Problems

Sat, 25 Jul 2026 11:18:34 +0200

Introduction to Debugging Connectivity Issues

I’ve seen this go wrong when dealing with connectivity issues on Linux systems: not knowing where to start. Two essential tools can save you a lot of time - resolvectl and ss. These tools help in identifying and troubleshooting DNS and routing problems. Let’s dive into a step-by-step guide on how to use them, focusing on practical examples and security considerations where relevant.

Understanding resolvectl

resolvectl is a command-line utility provided by systemd for querying and configuring the DNS resolver. It’s incredibly useful for controlling the DNS resolver, including setting DNS servers, querying the DNS cache, and configuring DNS over TLS (DoT) or DNS over HTTPS (DoH). To get started with resolvectl, you can use the following command to status the current DNS configuration:

Taming systemd Service Restart Policies to Prevent Unexpected Downtime

Wed, 22 Jul 2026 10:33:24 +0200

Introduction to systemd Service Restart Policies

I’ve seen many Linux systems go down due to poorly configured service restart policies. Systemd, being the core component of most modern Linux distributions, provides a robust way to manage system services, including their startup, runtime, and shutdown. One of the key features of systemd is its ability to manage service restart policies, which dictate how services should be handled in case of failures or crashes.

Taming Log Noise with journalctl: Filtering Out the Usual Suspects to Find Real Issues

Tue, 21 Jul 2026 11:01:50 +0200

Introduction to Log Noise

I’ve seen log noise become a significant issue for Linux administrators, making it tough to identify real problems in system logs. With the sheer amount of data generated by various system components, it’s easy to get lost. To tackle this, I usually start with journalctl, a powerful command-line utility that allows you to view and manage system logs. It’s part of the systemd suite and provides a flexible way to filter, prioritize, and analyze log messages.

Resolving DNS Quirks with systemd-resolved and Split DNSConfigs

Sun, 19 Jul 2026 09:59:52 +0200

Introduction to DNS Quirks and systemd-resolved

I’ve seen my fair share of DNS issues over the years, and in complex network environments, they can become a real headache. These problems often stem from misconfigured DNS servers, conflicting resolver settings, or the challenges of managing split DNS configurations. With systemd-resolved, managing DNS resolution has become more robust and secure. I’ve found it to be an essential tool in my Linux toolkit.

Taming Log Noise with jq and systemd Journal Filters

Sat, 18 Jul 2026 11:16:41 +0200

Introduction to Log Noise Reduction

I’ve seen this go wrong when working with Linux systems: log noise can be a significant issue, making it difficult to identify and troubleshoot problems. Log noise refers to the excessive amount of log data that is not relevant to the issue at hand. With the increasing complexity of systems, managing log noise has become crucial for efficient system administration. This is where people usually get burned - trying to sift through a sea of irrelevant log messages. The real trick is to use the right tools to tame log noise, and that’s where jq and systemd journal filters come in.

Taming Log Noise with systemd Journal Filters and Grep

Thu, 16 Jul 2026 09:20:19 +0200

Introduction to Log Noise

I’ve seen this go wrong when log noise gets out of hand - it’s like trying to find a needle in a haystack. As a Linux administrator, managing log noise is crucial for efficient system maintenance and debugging. In this article, we’ll explore how to tame log noise using systemd journal filters and grep.

Understanding Systemd Journal

Systemd journal is a centralized logging system that collects log messages from various system components, including systemd services, kernel messages, and application logs. The journal stores log messages in a binary format, which can be queried and filtered using the journalctl command. To view all log messages, you can use the following command:

Using resolvectl to Fix DNS Leaks and Improve Linux Network Privacy

Wed, 15 Jul 2026 08:20:33 +0200

Introduction to resolvectl

I’ve seen DNS leaks compromise network privacy time and again, so I’m always on the lookout for ways to tighten up my DNS configuration. As a Linux user, you’re likely familiar with the importance of DNS resolution for accessing websites and services. resolvectl is a powerful tool that comes with systemd, and it’s been a game-changer for managing DNS resolution on my systems.

What is resolvectl?

resolvectl is a command-line utility that allows you to control and configure DNS resolution on your system. It’s part of the systemd suite, which is widely used on modern Linux distributions. With resolvectl, you can set DNS servers, query DNS records, and troubleshoot DNS issues - all from the comfort of your terminal.

Troubleshooting Disk Usage Issues with btrfs Snapshots and systemd Timers

Tue, 14 Jul 2026 08:58:49 +0200

Introduction to btrfs Snapshots and systemd Timers

As someone who’s been running Linux systems for years, I’ve learned that managing disk usage and ensuring data integrity are crucial tasks. One effective way to achieve this is by utilizing btrfs snapshots and systemd timers. I’ve seen this go wrong when people don’t have a solid backup strategy in place, so it’s worth taking the time to set up. btrfs, a modern file system, offers advanced features like snapshotting, which allows you to create a read-only copy of your file system at a particular point in time. When combined with systemd timers, you can automate the creation and management of these snapshots, making it easier to troubleshoot disk usage issues and maintain a healthy system.

Using resolvectl to Debug Stubborn DNS Resolution Issues on Linux

Sun, 12 Jul 2026 08:51:43 +0200

Introduction to resolvectl

I’ve seen my fair share of DNS resolution issues on Linux, and one tool that’s always been a lifesaver is resolvectl. This command-line utility is part of the systemd suite, providing a comprehensive way to query and configure DNS settings on your system. When dealing with stubborn DNS problems, resolvectl is often the first tool I reach for. In this article, we’ll delve into its practical uses for debugging DNS issues, exploring its capabilities, and discussing how it can be used with other tools for a more robust troubleshooting experience.

Using systemd-resolved to Tame DNS Lookup Surprises on Multi-Homed Linux Machines

Sat, 11 Jul 2026 08:24:10 +0200

Introduction to systemd-resolved

I’ve seen this go wrong when working with multi-homed Linux machines: DNS lookup surprises can be a real pain. A multi-homed machine, with multiple network interfaces each connected to a different network, can lead to DNS resolution problems. The system may not know which interface to use for DNS lookups, causing all sorts of issues. This is where people usually get burned - trying to troubleshoot DNS problems without a clear understanding of how their system is configured. To address this issue, I usually start with systemd-resolved, a tool that can help manage DNS resolution and provide more predictable behavior.

Troubleshooting DNS Leaks with systemd-resolved and resolv.conf

Thu, 09 Jul 2026 08:15:21 +0200

Introduction to DNS Leaks

I’ve seen DNS leaks become a major concern for Linux users, especially those relying on systemd-resolved. A DNS leak occurs when your system sends DNS queries to an unintended DNS server, potentially exposing your browsing history and other sensitive information. In this article, I’ll share my experience and insights on how to identify and troubleshoot DNS leaks using systemd-resolved and resolv.conf.

Understanding systemd-resolved

Systemd-resolved is a powerful tool that provides DNS resolution. It acts as a local DNS resolver, caching DNS queries and responses to improve performance. By default, systemd-resolved uses the DNS servers provided by your network configuration, but you can configure it to use specific DNS servers. To check the current DNS servers used by systemd-resolved, run:

Extracting Valuable Data from Systemd Journal Logs with jq and grep

Wed, 08 Jul 2026 09:39:13 +0200

Introduction to Systemd Journal Logs

I’ve worked with Linux systems for years, and one of the most powerful tools at my disposal is systemd journal logs. These logs provide a centralized location for logging system events, making it easier to troubleshoot and diagnose issues. By default, systemd stores these logs in a binary format, which can make them a bit tricky to parse and analyze. However, with the help of tools like jq and grep, you can extract valuable data from these logs.

Troubleshooting Slow systemd Service Restarts with systemd-analyze

Sun, 05 Jul 2026 10:06:58 +0200

Introduction to systemd-analyze

I’ve seen my fair share of slow service restarts on Linux systems, and troubleshooting them can be a real challenge. That’s where systemd-analyze comes in - a powerful tool that helps analyze and troubleshoot systemd services. In this article, I’ll share how to use systemd-analyze to identify and potentially fix issues related to slow systemd service restarts.

Understanding systemd-analyze

The real trick is to understand what systemd-analyze can do for you. It’s a command-line tool that comes bundled with systemd, providing detailed information about the boot process, services, and their dependencies. With systemd-analyze, you can identify performance bottlenecks, slow services, and even issues with the system’s boot process. Don’t bother with manual logging and debugging - systemd-analyze can save you a lot of time and effort.

Resolving the "Unknown Host" Headache: Troubleshooting DNS Issues with systemd-resolved and resolv.conf

Wed, 01 Jul 2026 09:38:07 +0200

Introduction to DNS Troubleshooting

We’ve all been there - trying to access a website or service, only to be met with an “Unknown Host” error. These errors often stem from DNS issues, which can be frustrating to troubleshoot, especially when you’re not sure where to start. In my experience, the culprits are usually misconfigured resolv.conf files or problems with systemd-resolved. In this article, I’ll walk you through the practical aspects of troubleshooting DNS issues, focusing on systemd-resolved and resolv.conf, to help you resolve the “Unknown Host” headache efficiently.

Troubleshooting Btrfs Snapshot Overfill with systemd Timers and Log Rotation

Tue, 30 Jun 2026 09:01:31 +0200

Introduction to Btrfs Snapshots

I’ve worked with Btrfs for years, and its snapshot feature is one of my favorites. It allows you to create a read-only copy of a subvolume at a particular point in time, which is super useful for backups, testing, and rolling back changes. However, managing these snapshots can become a real pain, especially when dealing with a large number of them. In this article, I’ll show you how to troubleshoot Btrfs snapshot overfill issues using systemd timers and log rotation.

Troubleshooting Systemd Service Boot Delays with systemd-analyze

Mon, 29 Jun 2026 09:10:15 +0200

Introduction to Systemd Service Boot Delays

I’ve seen this go wrong when working with Linux systems managed by systemd - boot delays can be frustrating and impact system performance. In my experience, understanding how to troubleshoot these delays is crucial for maintaining efficient and reliable systems. Many Linux distributions, including Debian, Arch Linux, and OpenSUSE, rely on systemd for service management, so it’s essential to know how to use systemd-analyze to identify and resolve boot delays caused by systemd services.

Taming Log Rotation: Preventing Disk Space Issues with systemd-journald and Log File Management

Wed, 24 Jun 2026 11:55:31 +0200

Introduction to Log Rotation

Log rotation - it’s one of those tasks that’s easy to overlook, but can quickly become a major headache if you don’t stay on top of it. I’ve seen this go wrong when log files consume all the available disk space, bringing a system to its knees. With many Linux distributions now using systemd-journald as the default logging mechanism, understanding how to manage log rotation is more important than ever.

Taming systemd Service Restart Behavior: When to Use Restart, Retry, and Timeout Options

Mon, 22 Jun 2026 09:07:35 +0200

Introduction to systemd Service Restart Behavior

I’ve seen this go wrong when a service is not properly configured - systemd, the core component of most modern Linux distributions, is responsible for managing system services. One of its key features is the ability to automatically restart services that fail or terminate unexpectedly, controlled by the Restart directive in the service unit file. However, I’ve found that the Restart directive alone may not be sufficient to handle all scenarios, which is where the Retry and Timeout options come into play.

Using systemd-resolved with Multiple DNS Servers and Split Horizon DNS

Sat, 20 Jun 2026 08:55:08 +0200

Introduction to systemd-resolved

I’ve been using systemd-resolved for a while now, and I have to say, it’s a game-changer when it comes to managing DNS resolution on Linux systems. As of 2026, it’s become a crucial component in many Linux distributions, including Ubuntu, Debian, and Fedora. In this article, I’ll walk you through how to use systemd-resolved with multiple DNS servers and split horizon DNS, including some practical examples and troubleshooting notes.

Taming systemd Restart Policies to Prevent Service Chaos

Thu, 18 Jun 2026 10:48:52 +0200

Introduction to systemd Restart Policies

I’ve seen systemd restart policies go wrong when not properly configured, leading to a never-ending cycle of restarts without resolving the underlying issue. To avoid this chaos, it’s essential to understand how systemd manages system services, including starting, stopping, and restarting them as needed. The key to taming these restart policies lies in understanding how systemd service files work and how to configure them effectively.

Troubleshooting DNS Leaks with systemd-resolved and resolv.conf

Tue, 16 Jun 2026 10:58:16 +0200

Introduction to DNS Leaks

I’ve seen DNS leaks become a major concern for Linux users who care about their online privacy. Essentially, a DNS leak happens when your system’s DNS queries slip outside a secure tunnel - think VPN - and potentially expose your browsing history to prying eyes. Given the current online security landscape, being able to troubleshoot these leaks is more important than ever.

Understanding systemd-resolved

Systemd-resolved is a key system service in Linux that handles DNS resolution. It’s part of the systemd suite and is widely used in modern distributions like Ubuntu, Debian, and Fedora. One of the benefits of systemd-resolved is that it can be configured to use DNS over TLS (DoT) or DNS over HTTPS (DoH), which encrypts DNS queries and reduces the risk of leaks.

Resolving DNS Issues with resolvectl and systemd-resolved in a Home Network

Mon, 15 Jun 2026 09:54:03 +0200

Introduction to DNS Resolution

When setting up a home network, DNS resolution is one of those critical components that can be a real pain to troubleshoot. I’ve seen this go wrong when people are new to Linux, so let’s dive into how to use resolvectl to resolve DNS issues with systemd-resolved.

Understanding systemd-resolved

systemd-resolved is a system service that handles DNS resolution, among other network-related tasks. It’s part of the systemd suite and is widely used in many Linux distributions, including Ubuntu, Debian, and Fedora. The real trick is that systemd-resolved provides improved DNS security and better support for modern DNS protocols like DNS over TLS (DoT) and DNS over HTTPS (DoH). Don’t bother with trying to use it without understanding these benefits, as they’re a key part of what makes systemd-resolved so useful.

Troubleshooting Slow Network Connectivity with ss and resolvectl on Linux

Sun, 14 Jun 2026 08:16:06 +0200

Introduction to Network Troubleshooting

I’ve seen my fair share of slow network connectivity issues on Linux, and having the right tools at your disposal can make all the difference. Two tools that I rely on are ss and resolvectl, which can help you diagnose and troubleshoot network issues. In this article, we’ll explore how to use these tools to identify and potentially fix slow network connectivity problems.

Understanding `ss`

The ss command is a replacement for the traditional netstat command, and it provides more detailed information about network connections, including TCP, UDP, and Unix domain sockets. To get started with ss, you can use the following command to display all active connections:

Taming Disk-Hungry Logs with systemd's Persistent Journal and Log Rotation

Sat, 13 Jun 2026 11:46:55 +0200

Introduction to Log Management

I’ve seen log management become a major headache for many Linux administrators. Logs are essential for diagnosing issues, detecting security threats, and optimizing system performance, but they can grow rapidly and consume significant disk space. In practice, this can lead to performance issues and even system crashes. To avoid this, we can use systemd’s persistent journal and log rotation features.

Understanding systemd’s Journal

Systemd’s journal is a centralized logging system that collects log messages from various system components, including systemd services, kernel messages, and application logs. The real trick is to configure it to use persistent storage, so logs aren’t lost upon system reboot. By default, the journal stores log messages in a volatile storage area, which isn’t very useful for long-term log management.

Using systemd to Manage and Rotate Logs for Forgotten System Services

Fri, 12 Jun 2026 08:50:25 +0200

Introduction to Log Management with systemd

I’ve seen many Linux admins struggle with log management, especially when it comes to system services that are often overlooked. Systemd is a powerful system and service manager that provides a wide range of features, including process management, dependency handling, and log management. In this article, I’ll focus on using systemd to manage and rotate logs for system services.

Understanding systemd Logs

The real trick is to understand how systemd logs work. Systemd logs are stored in a binary format, which can be read using the journalctl command. This command provides a powerful way to filter, search, and manage system logs. By default, systemd stores logs in /var/log/journal, but this can be configured to use a different location. Don’t bother with trying to read the binary logs directly - just use journalctl.

Taming systemd Timer Services to Run Your Daily Backup at a Reasonable Hour

Tue, 09 Jun 2026 09:05:53 +0200

Introduction to systemd Timer Services

I’ve been using systemd timer services for years to schedule tasks on my Linux systems, and I have to say, they’re a game-changer. Most Linux distributions, including Debian, Arch Linux, and OpenSUSE, use systemd as their default init system, so it’s worth learning how to use them. In this article, I’ll show you how to use systemd timer services to run daily backups at a reasonable hour.

Taming systemd-resolved: Troubleshooting DNS leaks and resolving domain name surprises on Linux desktops and servers

Fri, 05 Jun 2026 10:35:34 +0200

Introduction to systemd-resolved

I’ve seen systemd-resolved become a crucial part of many Linux distributions, including Ubuntu, Debian, and Fedora, as of 2026. It’s designed to provide a robust and secure way to resolve domain names on Linux systems. However, like any complex system, it can sometimes behave unexpectedly, leading to DNS leaks and domain name resolution surprises.

Understanding DNS Leaks

A DNS leak occurs when your system sends DNS queries to an unintended DNS server, potentially revealing your browsing history and online activities to third parties. This can happen when your system is configured to use a specific DNS server, but systemd-resolved is not properly configured to respect this setting. Don’t bother with manually trying to diagnose DNS leaks - just use online tools such as dnsleaktest.com or ipleak.net to check for them.

Taming systemd-resolved: How to Configure DNS Settings for Split Horizon Environments

Thu, 04 Jun 2026 11:14:28 +0200

Introduction to systemd-resolved

I’ve seen many Linux admins struggle with configuring DNS settings for split horizon environments. systemd-resolved, a DNS resolver component of the systemd suite, can make life easier. In this article, I’ll walk you through how to configure DNS settings for split horizon environments using systemd-resolved.

Understanding Split Horizon Environments

Split horizon environments are network setups where multiple DNS servers provide different answers for the same domain name, depending on the client’s location or network. I’ve encountered this in organizations with multiple offices or data centers, where different DNS servers serve different locations. For example, a company with offices in the US and Europe might have two separate DNS servers, one for each region, providing different IP addresses for the same domain name.

Rescuing a Linux System Stuck in Emergency Mode: A Step-by-Step Recovery Guide

Wed, 03 Jun 2026 11:47:11 +0200

Introduction to Emergency Mode

I’ve seen this go wrong when a Linux system encounters a critical issue during boot - it may enter Emergency Mode. This mode provides a minimal environment for troubleshooting and recovery. With many Linux distributions, including Debian and Arch Linux, updating their boot processes to use systemd, understanding how to rescue a system stuck in Emergency Mode is crucial.

Identifying the Issue

To rescue a system in Emergency Mode, you first need to identify the cause of the issue. Don’t bother with guessing - check the system logs, which are usually available in the /var/log directory. I usually start with the journalctl command to view the system logs:

Troubleshooting Failed Mounts at Boot Time with systemd and fstab

Tue, 02 Jun 2026 09:15:54 +0200

Introduction to Troubleshooting Failed Mounts

When I’m dealing with a Linux system that won’t boot properly, one of the first things I check is the mount points. systemd and the /etc/fstab file are crucial in this process, but issues can still arise, leading to failed mounts and potential system instability. In this article, I’ll walk you through practical steps to troubleshoot and resolve failed mounts at boot time, focusing on systemd and fstab configurations.

Taming Disk-Hungry Logs with systemd-journald and logrotate

Mon, 01 Jun 2026 10:10:58 +0200

Introduction to Log Management

I’ve seen log files grow out of control and consume disk space, affecting system performance. To tame disk-hungry logs, I recommend using systemd-journald and logrotate. These tools help manage log data, making it easier to troubleshoot, debug, and perform security audits.

Understanding systemd-journald

systemd-journald is a system service that collects and stores log messages from various sources. It provides a centralized logging system, which I find more efficient than traditional text-based log files. To view log messages, use the journalctl command:

Rescuing a Broken Linux System with a systemd Emergency Mode Shell

Sun, 31 May 2026 08:26:25 +0200

Introduction to Emergency Mode

I’ve seen this go wrong when a Linux system encounters a critical issue during boot - it can be a real headache. But, thankfully, Linux has a built-in safety net called emergency mode. This mode kicks in when there’s a failed filesystem check or an inability to mount a necessary partition, providing a minimal environment for troubleshooting and repair. With the advancements in Linux, understanding how to use emergency mode is crucial for system administrators and users alike.

Troubleshooting DNS Leaks with systemd-resolved and resolv.conf on a Small Linux Server

Fri, 29 May 2026 08:36:48 +0200

Introduction to DNS Leaks

I’ve seen DNS leaks cause issues on even the most secure Linux servers. Ensuring your DNS setup is solid is crucial, and one common problem is a DNS leak, where your system inadvertently reveals your DNS queries to unauthorized parties. In this article, I’ll walk you through troubleshooting DNS leaks using systemd-resolved and resolv.conf on a Linux server.

Understanding systemd-resolved

systemd-resolved is a powerful tool that provides DNS resolution capabilities. It’s designed to be a caching, validating DNS resolver that can also handle DNSSEC validation. To check if systemd-resolved is running on your system, use the following command:

Debugging systemd Service Startup Failures with systemd-analyze and Journalctl

Thu, 28 May 2026 11:31:56 +0200

Introduction to Debugging systemd Services

I’ve seen this go wrong when you’re trying to troubleshoot issues with your Linux system - those pesky systemd services can be a real pain. They’re the backbone of your system, managing everything from network connections to system logging. Debugging these services can be a daunting task, especially for those new to Linux administration. Fortunately, systemd provides two powerful tools to help you diagnose and resolve issues: systemd-analyze and journalctl.

Taming Split DNS Chaos with systemd-resolved and Local Hostname Resolution

Wed, 27 May 2026 08:34:11 +0200

Introduction to Split DNS Chaos

I’ve seen this go wrong when working with multiple networks or self-hosted services: split DNS configurations can become a real headache. Luckily, many Linux distributions have started adopting systemd-resolved as the default DNS resolver, which makes managing split DNS scenarios much simpler. In this article, I’ll walk you through how to use systemd-resolved for local hostname resolution and taming that split DNS chaos.

Understanding systemd-resolved

The real trick is understanding how systemd-resolved works. It’s a systemd component that provides DNS resolution and caching, and it can be configured to use multiple DNS servers and handle split DNS scenarios with ease. To check if systemd-resolved is enabled on your system, run the following command:

Troubleshooting Slow DNS Lookups with systemd-resolved and resolvectl

Mon, 25 May 2026 08:57:59 +0200

Introduction to Troubleshooting Slow DNS Lookups

I’ve seen slow DNS lookups bring Linux systems to a crawl, and with our increasing reliance on online services, efficient DNS resolution is crucial. This article focuses on troubleshooting slow DNS lookups using systemd-resolved and resolvectl, which are integral to many modern Linux distributions.

Understanding systemd-resolved

systemd-resolved is a system service that provides DNS resolution, replacing traditional implementations like glibc’s resolver. It offers improved security, better DNSSEC handling, and efficient management of multiple DNS servers. To check if it’s running on your system, use:

Taming systemd-resolved: Avoiding DNS Leaks and Surprises with Split DNS Configurations

Sun, 24 May 2026 10:21:18 +0200

Introduction to systemd-resolved

I’ve worked with Linux systems for years, and one thing that’s become increasingly important is DNS resolution. systemd-resolved is a DNS resolver component of the systemd suite, designed to provide a flexible and secure way to resolve domain names. It was introduced in systemd version 216, released in 2015, and has since become a standard component in many Linux distributions. By default, systemd-resolved uses a split DNS configuration, which can sometimes lead to DNS leaks and unexpected behavior. I’ve seen this go wrong when a system has multiple network interfaces or connections, each with its own DNS resolver configuration.

Using systemd to Manage and Rotate Log Files Without Running Out of Disk Space

Sat, 23 May 2026 11:26:20 +0200

Introduction to Log Rotation with systemd

I’ve seen log files consume entire disks, bringing systems to a grinding halt. That’s why log rotation is crucial - it ensures your logs don’t get out of control. With systemd, you’ve got a robust mechanism for managing and rotating logs. In this article, I’ll dive into using systemd for log rotation, covering its benefits, configuration, and some practical examples.

Understanding systemd’s Role in Log Rotation

systemd’s journald is a game-changer for log management. It collects and stores log messages from various sources, including systemd services, kernel messages, and other system components. This centralized logging system makes it easier to manage and rotate logs. By leveraging systemd’s capabilities, you can configure log rotation to suit your specific needs, keeping your system stable and secure.

Taming systemd-resolved: Tips for Troubleshooting and Customizing DNS Resolution on Linux

Fri, 22 May 2026 10:04:50 +0200

Introduction to systemd-resolved

I’ve been using systemd-resolved for a while now, and I have to say, it’s a big improvement over traditional DNS resolvers. As of 2026, many Linux distributions, including Ubuntu, Debian, and Fedora, have adopted systemd-resolved as the default DNS resolver. While it offers several benefits, including improved security and performance, some users may encounter issues or require customization to suit their specific needs. Don’t bother with trying to disable it, though - it’s usually worth the effort to get it working right.

Using rsync and systemd to Automate Offsite Backups of Selected Config Files and User Data

Thu, 21 May 2026 10:49:01 +0200

Introduction to Automated Offsite Backups

As a Linux user, you’ve probably learned the hard way how important it is to protect your configuration files and user data from loss or corruption. I’ve seen this go wrong when a disk fails or a configuration change goes awry. One way to ensure the integrity of this data is to set up automated offsite backups. In this article, we’ll explore how to use rsync and systemd to create a reliable and efficient backup system.

Using jq to Parse and Manipulate JSON Logs from systemd-journald

Tue, 19 May 2026 08:58:34 +0200

Introduction to jq and systemd-journald

I’ve found that working with Linux systems often involves digging through logs to troubleshoot issues. systemd-journald is a key component in this process, collecting and storing log messages from various sources. Since these logs are often in JSON format, tools like jq become incredibly useful for parsing and manipulation. In this article, I’ll walk you through how to use jq to parse and manipulate JSON logs from systemd-journald.

Troubleshooting systemd Service Startup Failures with Dependency Ordering and Journalctl

Mon, 18 May 2026 10:30:29 +0200

Introduction to systemd Service Troubleshooting

I’ve seen this go wrong when services fail to start due to complex dependency ordering and logging issues. systemd, with its powerful tools for diagnosing problems, makes it easier to identify and fix these issues. In this article, we’ll focus on practical examples of using dependency ordering and journalctl to troubleshoot systemd service startup failures.

Understanding systemd Dependencies

systemd services are defined in unit files, typically located in /etc/systemd/system/ or /usr/lib/systemd/system/. These files specify the service’s dependencies, which are crucial for determining the order in which services start. Dependencies are defined using directives like Requires, Wants, Before, and After. For instance, a web server service might require the network service to be started before it can start itself.

Taming Log Rotation in systemd: A Practical Approach to Preventing Disk Bloat

Sun, 17 May 2026 10:14:26 +0200

Introduction to Log Rotation

I’ve seen this go wrong when log files grow out of control, filling up the disk and causing system instability. That’s why log rotation is a crucial aspect of Linux system maintenance. With many Linux distributions, including Debian and Arch Linux, adopting systemd as their default init system, understanding how to manage log rotation in a systemd environment is essential. In practice, this means getting familiar with systemd-journald, the component responsible for collecting and storing log messages.

Taming systemd Service Restart Behavior with RestartSec and TimeoutStartSec

Sat, 16 May 2026 08:13:26 +0200

Introduction to systemd Service Restart Behavior

I’ve seen this go wrong when a service fails and systemd keeps restarting it, causing more harm than good. To avoid this, it’s essential to understand how systemd handles service restarts. Systemd is a core component of most modern Linux distributions, responsible for managing system services, including their startup, runtime, and shutdown. One of the key aspects of systemd service management is its ability to automatically restart services that fail or terminate unexpectedly. However, this behavior can sometimes lead to unintended consequences, such as a service restarting indefinitely in a failed state. To mitigate this, systemd provides two important directives: RestartSec and TimeoutStartSec.

Systemd on Linux Security

Launch a temporary root process with `systemd‑run` instead of `sudo`

Run a single command as root with systemd‑run

What systemd‑run actually does

Fixing “Permission denied” on /tmp After a Kernel Update Removed the Sticky Bit

The sticky bit in a nutshell

What happened after the update

How to Get a Systemd Timer Back on Track After a Kernel Upgrade

Systemd timers can fall silent after a kernel upgrade

Using journalctl to Track Down the Hidden ‘eth0’ Carrier Lost Messages That Cause Network Flaps After a Kernel Upgrade

When a kernel upgrade silently breaks your network

Taming systemd Restart Policies to Prevent Service Deluge

Introduction to systemd Restart Policies

Recovering from a Failed Boot: Using systemd's Emergency Mode and Rescue Shell to Troubleshoot Initramfs Issues

Introduction to Emergency Mode and Rescue Shell

Taming Log Noise with journalctl: Filtering Out the Unimportant Stuff

Introduction to Log Noise

Troubleshooting systemd Service Restart Failures with Dependency Ordering and Retry Policies

Introduction to systemd Service Restart Failures

Understanding systemd Service Dependencies

Troubleshooting systemd Service Restart Loops with Dependency Ordering

Introduction to systemd Service Restart Loops

Using systemd to Manage and Rotate Log Files Without Running Out of Disk Space

Introduction to Log Rotation with systemd

Understanding systemd-journald

Taming systemd-resolved: Avoiding DNS Surprises on Multi-Network Linux Setups

Introduction to systemd-resolved

Configuring systemd-resolved

Taming Log Noise with jq and systemd Journal Filters

Introduction to Log Noise

Understanding systemd Journal

Using resolvectl to Diagnose and Resolve Stubborn DNS Issues on Linux

Introduction to resolvectl

Understanding DNS Resolution

Taming Systemd Service Restart Policies to Prevent Cascading Failures

Introduction to Systemd Service Restart Policies

Understanding Systemd Service Restart Policies

Taming Log Noise with jq and systemd Journal Filters

Introduction to Log Noise

Understanding systemd Journal

Taming log Noise with journalctl: Filtering Out the Chaff to Find Meaningful Errors

Introduction to Journalctl

Understanding Journalctl Basics

Debugging Connectivity Issues with resolvectl and ss: A Step-by-Step Guide to Identifying DNS and Routing Problems

Introduction to Debugging Connectivity Issues

Understanding resolvectl

Taming systemd Service Restart Policies to Prevent Unexpected Downtime

Introduction to systemd Service Restart Policies

Taming Log Noise with journalctl: Filtering Out the Usual Suspects to Find Real Issues

Introduction to Log Noise

Resolving DNS Quirks with systemd-resolved and Split DNSConfigs

Introduction to DNS Quirks and systemd-resolved

Taming Log Noise with jq and systemd Journal Filters

Introduction to Log Noise Reduction

Taming Log Noise with systemd Journal Filters and Grep

Introduction to Log Noise

Understanding Systemd Journal

Using resolvectl to Fix DNS Leaks and Improve Linux Network Privacy

Introduction to resolvectl

What is resolvectl?

Troubleshooting Disk Usage Issues with btrfs Snapshots and systemd Timers

Introduction to btrfs Snapshots and systemd Timers

Using resolvectl to Debug Stubborn DNS Resolution Issues on Linux

Introduction to resolvectl

Using systemd-resolved to Tame DNS Lookup Surprises on Multi-Homed Linux Machines

Introduction to systemd-resolved

Troubleshooting DNS Leaks with systemd-resolved and resolv.conf

Introduction to DNS Leaks

Understanding systemd-resolved

Extracting Valuable Data from Systemd Journal Logs with jq and grep

Introduction to Systemd Journal Logs

Troubleshooting Slow systemd Service Restarts with systemd-analyze

Introduction to systemd-analyze

Understanding systemd-analyze

Resolving the "Unknown Host" Headache: Troubleshooting DNS Issues with systemd-resolved and resolv.conf

Introduction to DNS Troubleshooting

Troubleshooting Btrfs Snapshot Overfill with systemd Timers and Log Rotation

Introduction to Btrfs Snapshots

Troubleshooting Systemd Service Boot Delays with systemd-analyze

Introduction to Systemd Service Boot Delays

Launch a temporary root process with `systemd‑run` instead of `sudo`

Run a single command as root with `systemd‑run`

What `systemd‑run` actually does

Understanding `ss`