I’ve found that working with Linux systems often involves digging through logs to troubleshoot issues. systemd-journald is a key component in this process, collecting and storing log messages from various sources. Since these logs are often in JSON format, tools like jq become incredibly useful for parsing and manipulation. In this article, I’ll walk you through how to use jq to parse and manipulate JSON logs from systemd-journald.
I’ve seen this go wrong when services fail to start due to complex dependency ordering and logging issues. systemd, with its powerful tools for diagnosing problems, makes it easier to identify and fix these issues. In this article, we’ll focus on practical examples of using dependency ordering and journalctl to troubleshoot systemd service startup failures.
systemd services are defined in unit files, typically located in /etc/systemd/system/ or /usr/lib/systemd/system/. These files specify the service’s dependencies, which are crucial for determining the order in which services start. Dependencies are defined using directives like Requires, Wants, Before, and After. For instance, a web server service might require the network service to be started before it can start itself.
I’ve seen this go wrong when log files grow out of control, filling up the disk and causing system instability. That’s why log rotation is a crucial aspect of Linux system maintenance. With many Linux distributions, including Debian and Arch Linux, adopting systemd as their default init system, understanding how to manage log rotation in a systemd environment is essential. In practice, this means getting familiar with systemd-journald, the component responsible for collecting and storing log messages.
[Read More]I’ve seen this go wrong when a service fails and systemd keeps restarting it, causing more harm than good. To avoid this, it’s essential to understand how systemd handles service restarts. Systemd is a core component of most modern Linux distributions, responsible for managing system services, including their startup, runtime, and shutdown. One of the key aspects of systemd service management is its ability to automatically restart services that fail or terminate unexpectedly. However, this behavior can sometimes lead to unintended consequences, such as a service restarting indefinitely in a failed state. To mitigate this, systemd provides two important directives: RestartSec and TimeoutStartSec.
I’ve seen this go wrong when you’re in the middle of a critical project and a package update breaks a dependency, causing system instability. In Debian-based systems, apt-mark is a useful tool for pinning packages and avoiding this kind of chaos. It’s not a silver bullet, but it can help prevent packages from being automatically removed or upgraded, which can cause conflicts.
apt-mark is a command-line tool that allows you to mark packages as automatically installed or manually installed. This can be a lifesaver when you need to prevent a package from being automatically upgraded to a newer version that may cause conflicts. Don’t bother with trying to manually manage dependencies - apt-mark makes it easy to pin packages to a specific version.
Securing SSH access is a no-brainer for any Linux administrator. One effective way to harden SSH access is by using mandatory SSH keys and disabling password authentication. I’ve seen this go wrong when people don’t take the time to set it up properly, so let’s walk through the process.
SSH keys are a pair of cryptographic keys used for authentication. The private key is stored on the client machine, while the public key is stored on the server. When a user attempts to connect to the server, the client uses the private key to encrypt a message, which is then decrypted by the server using the public key. If the decryption is successful, the user is granted access.
[Read More]As a Linux user, you’re probably no stranger to Secure Shell (SSH) and its importance in securely accessing remote servers. With the rise of SSH server certificates, I’ve seen this go wrong when not properly verified - it’s crucial to ensure the authenticity and integrity of your SSH connections. In this article, we’ll explore how to use OpenSSL to verify SSH server certificates on Linux clients.
[Read More]I’ve seen many Linux systems compromised due to weak SSH security, so I want to share my approach to hardening SSH access. As of 2026, SSH remains a vital tool for system administration, development, and self-hosting, but its widespread use also makes it a common target for brute-force attacks and unauthorized access attempts. To mitigate these risks, I recommend using Fail2Ban and Public Key Authentication to secure SSH on Debian-based systems.
[Read More]I’ve seen many Linux admins struggle with managing SSH keys for their users. OpenSSH certificates are a game-changer here - they offer a more secure and convenient way to manage access to your Linux servers. By using certificates, you can avoid the hassle of managing individual SSH keys for each user and reduce the risk of key compromise.
To get started with OpenSSH certificates, you need to set up a Certificate Authority (CA). The CA will be used to sign and verify the certificates. I usually start with generating a CA key pair using the following command:
[Read More]Securing your SSH server is one of the most critical tasks as a Linux administrator. I’ve seen this go wrong when systems are left vulnerable to unauthorized access. The recent high-profile vulnerabilities discovered in popular SSH implementations in 2025 serve as a reminder to keep your SSH server up to date and configured securely. In this article, we’ll discuss practical steps to harden your SSH server without making it overly restrictive.
[Read More]