Linux Security

Why journald can eat your SD card

Running a Raspberry Pi on an 8 GB SD card is a rite of passage for home labs, IoT experiments, and low‑cost servers.
On Pi OS the default is to keep systemd‑journald persistent, so every log line ends up on that tiny flash drive.
In practice the Pi runs a handful of services—networking, SSH, Docker, custom scripts—that churn out logs faster than the card can keep up.
The result? The SD card fills up, you can’t install updates, and the OS itself starts to feel cramped.

Monitoring /home Usage with a Systemd Timer

A home server rarely needs a full‑blown monitoring stack.
If you just want a quick alert when /home fills up, a tiny systemd service plus curl will do the trick. The example below keeps the footprint minimal, respects security best practices, and uses the official Telegram Bot API.

1. Create a Bot and Grab Credentials

1. Talk to BotFather on Telegram and create a bot.
2. Note the bot token (e.g. 123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11).
3. Find your chat ID by sending a message to the bot and querying https://api.telegram.org/bot<token>/getUpdates.
4. Store both values in a read‑only file:

sudo mkdir -p /etc/telegram-alert
sudo tee /etc/telegram-alert/credentials <<EOF
TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
CHAT_ID=987654321
EOF
sudo chmod 600 /etc/telegram-alert/credentials

2. Service Unit – home-usage-alert.service

# /etc/systemd/system/home-usage-alert.service
[Unit]
Description=Send Telegram alert when /home exceeds 90%% usage
After=network.target

[Service]
Type=oneshot
EnvironmentFile=/etc/telegram-alert/credentials
ExecStart=/usr/bin/bash -c '
  USAGE=$(df --output=pcent /home | tail -1 | tr -d "%")
  if [ "$USAGE" -ge 90 ]; then
    curl -s -X POST https://api.telegram.org/bot$TOKEN/sendMessage \
      -d chat_id=$CHAT_ID \
      -d text="⚠️ /home is at ${USAGE}%% capacity. Free space needed." \
      -H "Content-Type: application/x-www-form-urlencoded"
  fi
'
PrivateTmp=yes
ProtectSystem=full
RestrictAddressFamilies=AF_INET AF_INET6
Restart=no

[Install]
WantedBy=multi-user.target

Security notes

Why the “REMOTE HOST IDENTIFICATION HAS CHANGED!” error appears

The message “REMOTE HOST IDENTIFICATION HAS CHANGED!” shows up every time you try to SSH into a host that reports a different key than the one you stored in ~/.ssh/known_hosts.
It’s a sanity check that protects you from man‑in‑the‑middle attacks, but it can also be a pain when you’re just redeploying a server or spinning up a VM that got a new IP.

When a host rebuilds, its SSH key changes

A fresh install or a system rebuild swaps the host’s RSA/ED25519 key pair.
If you still have the old key in ~/.ssh/known_hosts, the next SSH connection will trigger a “WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!” message.
That warning is a safety feature, but it also blocks automation and forces users to manually edit the file.

The quickest, safest way to clean up stale entries is to let OpenSSH do the work with ssh-keygen -R. The command removes every key that matches the supplied hostname or IP, regardless of whether the entry is hashed or plain.

After a kernel upgrade a service that used to run fine suddenly dies.
The first instinct is to blame the new kernel, but the real problem is often a missing module, a changed sysctl, or a subtle API change.
journalctl is the first tool you should use to trace the exact moment the service stops, what the kernel was doing, and whether the crash was caused by the upgrade or by something else.

Enabling systemd‑resolved is the go‑to way to get modern DNS features on recent distributions, but it can bite you when you forget a tiny detail.
The most common symptom: the entries in /etc/hosts no longer resolve, or worse, the system starts hitting a public DNS server that ignores the file entirely.
Usually it’s a mis‑configured NSS order, a stale symlink, or a permission slip.
Below is a practical, hands‑on walk‑through that covers the most common causes, shows how to diagnose each one, and gives you a clean fix that keeps your system secure and predictable.

Why /boot/efi Fails After a Kernel Upgrade

When you pull a new kernel into the system, the initramfs gets rebuilt and the bootloader is updated. On UEFI machines that keep the EFI system partition in its own /boot/efi, the kernel mounts that partition right after booting to read EFI variables and the boot‑loader configuration. A mis‑configured /etc/fstab, a missing vfat module, or a stale initramfs can bite you early, dropping you into an emergency shell before the kernel even gets a chance to run. You’ll usually see something like:

When a new NVMe sticks, the boot can stall at “Loading initramfs”

Adding a second NVMe drive to an existing Ubuntu 24.04 installation is a quick way to get more storage or carve out a dedicated data partition.
Most of the time the kernel pulls the NVMe driver on its own, but a handful of mis‑configurations can make the system hang in the initramfs shell.
Typical symptoms:

• The screen shows “Loading initramfs …” and then stops.
• The prompt switches to a root shell (root@hostname:/#).
• Nothing else happens until you manually exit or reboot.

Below is a hands‑on walk‑through of the most common culprits and how to fix them. I’ll assume you have a working Ubuntu 24.04 install and a spare NVMe device that you just plugged in.

Run a single command as root with systemd‑run

When you just need to fire up one command as root and then drop back, sudo is the go‑to.
But if your distro already runs systemd, systemd‑run can be a leaner, more isolated way to do it—especially for short‑lived tasks.

What systemd‑run actually does

systemd‑run spawns a transient unit (either a scope or a service) that inherits the caller’s privileges and then elevates them to root when you add --user or --scope with --user.
Unlike sudo, the unit vanishes as soon as the command finishes, and you can sprinkle a handful of systemd properties on it to tighten the environment.

When you fire up a heavy build, a data‑scraping job, or a local web server inside a tmux session, the usual pattern is to detach and log out. tmux will happily keep the session alive as long as your login shell stays around, but as soon as you hit logout or PAM times you out, the whole tree dies. Wrapping tmux in a systemd unit makes it immune to those surprises and gives you the same tooling you already use for other services—logging, restart policies, limits, and isolation.