Tracing a Sudden Drop‑in‑Memory Spike with journalctl and systemd‑coredump logs -

Sudden Memory Spike? Use journalctl + systemd‑coredump to Pin It Down

When a server’s RAM jumps from 30 % to 90 % in a matter of minutes, the first instinct is to kill processes, restart services, or add more memory. Those are quick fixes, but they don’t tell you why the spike happened. On a system that runs systemd, the combination of journalctl and systemd‑coredump gives a precise, timestamp‑aligned view of what went wrong. This post walks through a real‑world scenario, shows how to extract the relevant logs, and explains how to interpret the data so you can fix the root cause and harden the system against future incidents.

1. Recreate the Problem in a Controlled Testbed

For reproducibility we’ll use a minimal Debian 12 VM:

2 GB RAM, 2 vCPU
systemd 252 (default in Debian 12)
systemd‑coredump enabled (default)
journalctl configured to keep logs for 7 days

# Install the minimal set
apt-get update && apt-get install -y systemd systemd-coredump

Trigger a memory leak in a tiny C program that allocates 10 MiB in a loop until it crashes:

// leak.c
#include <stdlib.h>
#include <unistd.h>

int main(void) {
    while (1) {
        void *p = malloc(10 * 1024 * 1024); // 10 MiB
        (void)p;
        sleep(1);
    }
}

Compile and run it in the background:

gcc -O2 -Wall leak.c -o leak
./leak &

After a few minutes the process will eat almost all RAM and the kernel will start killing it with OOM_KILLER. In a real system, the spike might happen during a web request, a database query, or a cron job. The key is that the spike is sudden and the system’s memory pressure is visible in top or htop.

2. Capture the Timeline with `journalctl`

journalctl stores all system logs in a binary format that preserves timestamps, PIDs, and priority levels. The first step is to isolate the period around the spike.

# Show the last 5 minutes of logs, sorted by time
journalctl -S -5min -o short-iso

The -S flag accepts a relative time (-5min), and -o short-iso gives a concise, sortable output. Look for high‑priority messages (ERR, CRIT) that mention memory, OOM, or coredumps.

2026-10-29T14:12:07.123456+00:00 kernel: Out of memory: Kill process 1234 (leak) score 987 or sacrifice child
2026-10-29T14:12:07.123789+00:00 systemd[1]: leak.service: Failed with result 'killed'.

The kernel line tells us the OOM killer chose our leak process (PID 1234). The systemd line shows the service unit that ran it. If the process was launched by a user or a script, the log will still reference the unit that owns it.

Filtering by Unit

If you know the unit name (e.g., myapp.service), filter directly:

journalctl -u myapp.service -S -5min

This cuts out the noise and focuses on the relevant process.

Persisting the Log Slice

For long‑term analysis, export the slice to a file:

journalctl -u myapp.service -S -5min > /tmp/myapp-oom.log

You can later grep or parse this file with tools like awk or jq (if you convert to JSON with -o json-pretty).

3. Locate the Core Dump

When systemd‑coredump is enabled, the kernel writes a core file to /var/lib/systemd/coredump/. The file name encodes the PID, UID, and timestamp. Use journalctl to find the exact core file name:

journalctl -u myapp.service -S -5min | grep -i core

Typical output:

2026-10-29T14:12:07.124000+00:00 systemd-coredump[1]: Core file written to /var/lib/systemd/coredump/core.1234.1000.2026-10-29T14:12:07.123Z

The core.1234.1000 part means:

1234 – PID
1000 – UID (usually the user who started the process)

If you see multiple core files, pick the one that matches the timestamp of the OOM event.

Verify the Core File

ls -l /var/lib/systemd/coredump/core.1234.1000.*

The file size should be large (the leaked memory). If the file is missing, check the systemd‑coredump configuration:

cat /etc/systemd/coredump.conf

Key directives:

Storage=external – writes to /var/lib/systemd/coredump/
Compress=yes – compresses the core file (adds .gz)

If compression is on, you’ll see a .gz suffix.

4. Inspect the Core Dump with `gdb`

gdb can read the core file and the binary that produced it. This lets you see the stack trace at the point of failure.

gdb -c /var/lib/systemd/coredump/core.1234.1000.gz ./leak

If the core is compressed, gdb will automatically decompress it. Once inside gdb, run:

(gdb) bt

The backtrace will show the function chain that led to the crash. In our leak example, you’ll see a loop that never returns, confirming the memory leak.

For more complex binaries, you may need symbols. If the binary is stripped, install the debug symbols package (e.g., apt-get install myapp-dbgsym) or use objdump -d to disassemble.

5. Correlate with System Metrics

While the core dump tells you what crashed, you also want to see how the system behaved. journalctl can be combined with systemd-analyze and systemd-cgtop to get a holistic view.

# Memory usage over time
journalctl -b -o short-iso | grep -i memory

The -b flag limits to the current boot. Look for MemoryPressure events:

2026-10-29T14:12:07.124000+00:00 systemd[1]: MemoryPressure: 90%

systemd-cgtop shows the cgroup that consumed the most memory:

systemd-cgtop -t 5

The -t 5 flag updates every 5 seconds. Observe the cgroup of myapp.service before and after the spike.

6. Common Pitfalls and How to Avoid Them

Symptom	Likely Cause	Fix
Core file missing	`systemd‑coredump` disabled or `Storage=none`	`systemctl edit systemd-coredump` → `Storage=external`
Core file too small	Process killed by OOM before core written	Increase `coredump_filter` or set `SystemMaxUse=` in the unit
`journalctl` shows no OOM message	OOM killer suppressed by `OOMScoreAdjust`	Check `/proc/<pid>/oom_score_adj`
Core file compressed, `gdb` fails	`gdb` older than 7.11	Upgrade `gdb` or decompress manually (`gunzip`)
Multiple cores for same PID	Process forked; each child got a core	Use `coredump_filter` to include only parent (`0x7f`)

7. Security Considerations

Core Dump Sensitivity
Core files can contain plaintext passwords, API keys, or other secrets. Restrict access:

chmod 600 /var/lib/systemd/coredump/core.*

Or configure systemd-coredump to store cores in a secure directory:

# /etc/systemd/coredump.conf
Storage=external
Compress=yes
ExternalPath=/var/lib/systemd/coredump
ExternalPermissions=0600

Audit Log Retention
Keep logs long enough to investigate but not forever. systemd-journald can be configured to rotate after a size limit:
```
# /etc/systemd/journald.conf
SystemMaxUse=1G
SystemKeepFree=200M
```
Avoid Privileged Core Dumps
If a process runs as root, its core can reveal kernel memory. Disable core dumps for privileged services:
```
# /etc/systemd/system/myapp.service.d/override.conf
[Service]
ProtectKernelTunables=yes
```

8. Automating the Investigation

For production systems you rarely want to run these commands manually. Create a small helper script:

#!/usr/bin/env bash
# /usr/local/bin/oom-analyze
set -euo pipefail

UNIT=${1:-myapp.service}
TIME=${2:-5min}

# 1. Grab journal slice
journalctl -u "$UNIT" -S "$TIME" > /tmp/oom-journal.log

# 2. Find core file
CORE=$(journalctl -u "$UNIT" -S "$TIME" | grep -i core | awk '{print $NF}')
if [[ -z $CORE ]]; then
    echo "No core found"
    exit 1
fi

# 3. Show backtrace
gdb -batch -ex 'bt' -c "$CORE" "$(readlink -f /proc/$(echo $CORE | cut -d. -f2)/exe)" > /tmp/oom-backtrace.txt

# 4. Print summary
echo "=== OOM Event ==="
cat /tmp/oom-journal.log
echo
echo "=== Backtrace ==="
cat /tmp/oom-backtrace.txt

Run it whenever you see a memory spike:

oom-analyze myapp.service 10min

The script outputs a concise report that can be emailed to the dev team or stored in a ticketing system.

9. Real‑World Example: A Web Service Crash

Imagine a Node.js API that suddenly consumes 80 % of RAM after a heavy query. The steps above translate as:

Check the journal for OOM_KILLER entries and the service name (node.service).
Locate the core in /var/lib/systemd/coredump/.
Run node --inspect with gdb to get a stack trace.
Node’s core dumps are not as useful as native binaries, but you can still see the JavaScript call stack if you enable node --abort-on-uncaught-exception.
Inspect memory usage with systemd-cgtop to see if a particular cgroup (e.g., node.service.slice) is out of control.
Apply a fix: add a memory limit in the unit file:
```
[Service]
MemoryMax=512M
```
This forces the OOM killer to act before the system becomes unstable.

10. When the OOM Killer Isn’t the Culprit

Sometimes the spike is caused by a kernel bug or a device driver, not a userland process. In those cases:

Look for kernel: messages about bad page, DMA, or device errors.
Enable debugfs and kmsg to capture low‑level events.
Use perf or ftrace to trace memory allocations.

If you suspect a kernel issue, report it to the upstream maintainers. For Debian, file a bug against linux-image-<version>; for Fedora, use bugzilla.redhat.com.

11. Post‑Mortem Checklist

After you’ve identified the root cause, run through this checklist:

Patch the offending code or upgrade the package.
Add limits (MemoryMax, CPUQuota) to the unit.
Configure core dump retention to avoid disk fill.
Enable audit for the unit (systemd-analyze plot can help visualize resource usage over time).
Document the incident in your knowledge base with the exact journal snippet and backtrace.

12. TL;DR

Find the OOM event with journalctl -S -5min.
Locate the core in /var/lib/systemd/coredump/.
Inspect the core with gdb -c core binary.
Correlate with systemd-cgtop and systemd-analyze.
Secure core files and logs.
Automate the process with a small helper script.

With these steps, you turn a mysterious memory spike into a concrete, reproducible problem that can be fixed once and for all.