Using systemd‑cgtop to Spot the Service Eating All Your Memory -

Quick Memory Hunt with systemd‑cgtop

When a server starts to feel sluggish, most folks jump straight to CPU or disk I/O. Memory is the silent saboteur that slips under the radar until it’s too late. On modern Linux, every service lives in a cgroup, and systemd‑cgtop gives you a live, per‑service view of how that cgroup is eating RAM. Unlike top or ps, it aggregates the numbers for you, so you can spot the offender in a glance.

What is systemd‑cgtop?

systemd‑cgtop is a lightweight, real‑time monitor that reads the cgroup hierarchy and prints the top consumers of CPU, memory, and I/O. It works with both cgroup v1 and v2. The output refreshes every two seconds by default, but you can tweak the interval with -t.

sudo systemd-cgtop

Typical output:

  CPU%   MEM%   I/O%  UNIT
  0.00    0.00    0.00  system.slice
  0.00    0.00    0.00  user.slice
  0.00    0.00    0.00  dbus.service
  0.00    0.00    0.00  sshd.service

The columns show the percentage of the total resource pool used by each cgroup, and the UNIT column lists the systemd unit that owns the cgroup. Because the numbers are cumulative, a service that spawns many short‑lived processes may still look light on memory even though each child briefly hogs a chunk.

Narrowing the view to a single service

If you already suspect a particular service, just filter on it:

sudo systemd-cgtop --unit nginx.service

That pulls the entire hierarchy under nginx.service, which is handy when a host runs dozens of services and you want to focus on one.

Interpreting the memory column

The memory column is cumulative across all processes in the cgroup. It includes:

Resident Set Size (RSS) of each process.
Shared memory mapped by the cgroup (e.g., shared libraries).
Kernel bookkeeping for the cgroup itself (e.g., memory‑cgroup bookkeeping).

Because of that, a service that loads a big shared library—think postgresql or a large Java runtime—might look like it’s using more memory than the actual process footprint. If you need a finer granularity, pair systemd-cgtop with ps:

ps -o pid,comm,rss,cmd -C nginx

The rss column shows the resident memory of each process in kilobytes.

Spotting a memory hog: a practical workflow

Initial scan
```
sudo systemd-cgtop -t 1
```
Watch the MEM% column. If a service jumps to 30–50 % of total RAM, it’s worth digging deeper.
Confirm the culprit
```
sudo systemd-cgtop --unit nginx.service
```
If the memory stays high, the service is probably the source.
Dive deeper
```
ps -o pid,comm,rss,cmd -C nginx
```
Look for unusually high RSS values or a process that never terminates.
Check logs
```
journalctl -u nginx.service --since "1 hour ago"
```
Errors or repeated restarts can explain a memory leak.
Apply limits (if needed)
Edit /etc/systemd/system/nginx.service.d/override.conf:
```
[Service]
MemoryMax=512M
```
Reload systemd and restart:
```
sudo systemctl daemon-reload
sudo systemctl restart nginx.service
```
The MemoryMax directive caps the cgroup’s memory usage, and the kernel will kill the offending process if the limit is exceeded.

Security considerations

A runaway service can be a DoS vector. Even a well‑written service can exhaust RAM if it’s misconfigured—say, unlimited worker processes or a memory‑leaking module. Setting MemoryMax or MemoryHigh enforces a hard boundary, protecting the rest of the system.

Cgroup isolation is a core part of systemd’s security model. If a service is compromised, its memory usage stays confined to its cgroup, preventing it from affecting other services. Monitoring with systemd-cgtop keeps that isolation in check.

Trade‑offs: cgtop vs. other tools

Tool	Strength	Limitation
`systemd-cgtop`	Real‑time per‑service view; integrates with systemd units	Only shows aggregated cgroup data; not per‑process
`top` / `htop`	Per‑process detail; interactive	Requires manual filtering for services
`ps`	Snapshot of processes	No live refresh; needs manual aggregation
`systemd-analyze blame`	Shows startup time per unit	Static snapshot; no memory data

For a quick memory hunt, systemd-cgtop is the fastest. For deeper forensic analysis, combine it with ps and journalctl.

Common pitfalls and troubleshooting

Symptom	Likely cause	Fix
`systemd-cgtop` shows 0 % memory for all services	Kernel missing memory cgroup support	Reboot with `CONFIG_MEMCG=y` or enable `memory` controller in `/etc/fstab`
Memory percentages don’t add up to 100 %	Some memory is allocated to the kernel or shared libraries	Use `ps` for per‑process RSS
`systemd-cgtop` hangs or is slow	High number of cgroups (e.g., many containers)	Use `–unit

linux open-source