Systemd

Why /tmp on a tmpfs is a lifesaver for log‑heavy workloads

If a service starts spamming logs, a full /tmp can bring the whole system to a halt. On low‑RAM servers or in containers where /tmp lives on a regular disk partition, the problem is even worse. Mounting /tmp as a tmpfs keeps it in memory, guarantees instant cleanup, and stops disk‑space exhaustion. Below is a practical, step‑by‑step guide that covers the trade‑offs and security notes you’ll need to keep in mind.

Why the “Broken Packages” Hang is a Common Pain Point

When I run a fresh upgrade from Ubuntu 22.04 LTS to 24.04 LTS on a home server, the first thing that can bite me is the infamous “Broken Packages” dialog that stalls the upgrade process. It isn’t a bug in the installer; it’s the package manager trying to juggle a huge shift in dependencies between releases. The result is a long pause—sometimes hours—before the system finally moves on. For a server that should stay online, that pause is unacceptable.

When systemd Unit Files Don’t Pick Up Changes

You tweak a unit file, hit systemctl, and nothing happens.
That’s the classic “I changed the file, but systemd still runs the old version” problem.
The daemon is still holding the old config in memory. systemd keeps a cache of unit files; it only rereads them after a daemon‑reload. If you skip that step, the changes stay invisible until you reboot or reload manually.

Monitoring /home Usage with a Systemd Timer

A home server rarely needs a full‑blown monitoring stack.
If you just want a quick alert when /home fills up, a tiny systemd service plus curl will do the trick. The example below keeps the footprint minimal, respects security best practices, and uses the official Telegram Bot API.

1. Create a Bot and Grab Credentials

1. Talk to BotFather on Telegram and create a bot.
2. Note the bot token (e.g. 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11).
3. Find your chat ID by sending a message to the bot and querying https://api.telegram.org/bot<token>/getUpdates.
4. Store both values in a read‑only file:

sudo mkdir -p /etc/telegram-alert
sudo tee /etc/telegram-alert/credentials <<EOF
TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
CHAT_ID=987654321
EOF
sudo chmod 600 /etc/telegram-alert/credentials

2. Service Unit – home-usage-alert.service

# /etc/systemd/system/home-usage-alert.service
[Unit]
Description=Send Telegram alert when /home exceeds 90%% usage
After=network.target

[Service]
Type=oneshot
EnvironmentFile=/etc/telegram-alert/credentials
ExecStart=/usr/bin/bash -c '
  USAGE=$(df --output=pcent /home | tail -1 | tr -d "%")
  if [ "$USAGE" -ge 90 ]; then
    curl -s -X POST https://api.telegram.org/bot$TOKEN/sendMessage \
      -d chat_id=$CHAT_ID \
      -d text="⚠️ /home is at ${USAGE}%% capacity. Free space needed." \
      -H "Content-Type: application/x-www-form-urlencoded"
  fi
'
PrivateTmp=yes
ProtectSystem=full
RestrictAddressFamilies=AF_INET AF_INET6
Restart=no

[Install]
WantedBy=multi-user.target

Security notes

Run a single command as root with systemd‑run

When you just need to fire up one command as root and then drop back, sudo is the go‑to.
But if your distro already runs systemd, systemd‑run can be a leaner, more isolated way to do it—especially for short‑lived tasks.

What systemd‑run actually does

systemd‑run spawns a transient unit (either a scope or a service) that inherits the caller’s privileges and then elevates them to root when you add --user or --scope with --user.
Unlike sudo, the unit vanishes as soon as the command finishes, and you can sprinkle a handful of systemd properties on it to tighten the environment.

The sticky bit in a nutshell

The sticky bit (+t) on a directory means that only the file’s owner, the directory’s owner, or root can delete or rename files inside it. /tmp is traditionally set to 1777 (octal) – owner root, group root, permissions rwxrwxrwt. That lets any user create files, but stops them from messing with each other’s data.

What happened after the update

Some kernel releases changed the default mount options for tmpfs or how tmpfs‑based /tmp is handled. If /tmp is mounted as tmpfs without the sticky bit, the kernel quietly clears the +t flag. The directory stays world‑writable, but the protection that stops users from deleting each other’s files disappears. Applications that expect the sticky bit now refuse to write, because they see the directory as insecure.

Systemd timers can fall silent after a kernel upgrade

When a new kernel lands, the kernel‑specific modules and initramfs changes can break the ExecStartPre or ExecStart scripts that a timer relies on.
The result? A timer that never fires, or that fires with the wrong environment.

Below is a step‑by‑step guide to diagnose and restore a broken timer, with security‑aware notes and trade‑offs you’ll run into in a production or homelab setup.

When a kernel upgrade silently breaks your network

Kernel upgrades are usually painless, but on a few systems they can trigger a cascade of “carrier lost” messages that make the interface drop and come back up repeatedly. The messages are buried deep in the system journal, so you may not notice them until the network flaps start causing outages. This post shows how to locate those hidden messages with journalctl, correlate them with real‑world symptoms, and apply a fix that keeps the interface stable.

Introduction to systemd Restart Policies

I’ve seen this go wrong when a service is misconfigured and ends up consuming all your system resources. Systemd is a core component of most modern Linux distributions, responsible for managing system services. One of its key features is the ability to automatically restart services that fail or exit unexpectedly. However, if not properly configured, this can lead to a “service deluge” where a failing service is repeatedly restarted.

Introduction to Emergency Mode and Rescue Shell

When your Linux system fails to boot, it can be frustrating - especially if you’re not familiar with the troubleshooting process. I’ve seen this go wrong when people don’t know where to start. Luckily, systemd’s Emergency Mode and Rescue Shell are here to help. In this article, we’ll dive into how to use these tools to troubleshoot initramfs issues and get your system up and running again.