← All posts Deep dive

Your disk is at 89%. Here's how the pros find what's eating it.

The Tryssh team · July 20, 2026

Every server dies the same way eventually: quietly, at 2am, when a disk hits 100% and writes start failing. The database can't checkpoint. The app can't log. Sometimes you can't even SSH in properly because the box can't write to /var/log/auth.log.

The good news is that disk-full is the most solvable incident there is. The space went somewhere, and Linux will tell you exactly where — if you ask in the right order.

Start wide, then drill

Don't guess. Don't start deleting things you remember being big. Run the sequence:

df -h                                    # which filesystem is actually full?
du -xh --max-depth=1 / 2>/dev/null | sort -rh | head
du -xh --max-depth=1 /var 2>/dev/null | sort -rh | head

The -x flag matters — it keeps du on one filesystem so you don't waste ten minutes scanning an NFS mount. Drill one level at a time until you hit the directory that owns the gigabytes. Nine times out of ten it's one of these:

  • journald. journalctl --disk-usage will casually admit to 4GB of logs you've never read.
  • Docker. Images, stopped containers, and build cache. du -sh /var/lib/docker to confirm.
  • Rotated logs that never got deleted. ls -lhS /var/log and look for .gz files from months ago.
  • Package caches. du -sh /var/cache/apt/archives — every deb you ever installed, kept forever.
  • Old kernels filling /boot on long-lived Ubuntu boxes.

When df and du disagree, a process is lying to you

This is the one that burns people. df says the disk is full; du adds everything up and finds 20GB missing. Neither is broken.

Someone — maybe you, maybe a cleanup script — deleted a log file that a process still had open. The directory entry is gone, so du can't see it. But the kernel keeps the blocks allocated until the last file descriptor closes, so df still counts them. The file is a ghost: taking up space, invisible on disk.

Find the ghosts:

lsof +L1

Every row is a deleted file some process is still holding. The fix is not another rm — there's nothing left to remove. Restart the process, or truncate through procfs if a restart is too expensive. And learn the underlying lesson: never rm an active log file. Truncate it (> app.log) or let logrotate do its job. Deleting it frees nothing and creates a ghost.

The other invisible failure: inodes

Occasionally df -h shows plenty of space and writes still fail with "No space left on device." Check inodes:

df -i

If IUse% is at 100%, you don't have a big-file problem — you have a millions-of-tiny-files problem. PHP session files, mail queues, container layers, some cron job that's been writing one file per minute since 2024. Count files per directory to find the nest:

find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head

Cleanups that are safe — and the ones that bite

The safe list, in the order I'd run them:

journalctl --vacuum-time=7d          # keep a week of journal
apt clean && apt autoremove --purge  # caches + old kernels (keeps current)
docker container prune -f            # stopped containers only
docker image prune -f                # dangling images only

Now the bites. Two commands look similar to the ones above and have ended careers:

docker system prune -a --volumes removes named volumes that no running container references. If your database container happens to be stopped for maintenance, its data volume is "unused." People have pruned production databases this way. Run docker volume ls first, and handle volumes by hand — Docker has no dry-run.

Manually deleting database WAL files. Postgres's pg_wal at 50GB is a symptom (usually a dead replication slot), not a folder to empty. rm in there cuts the WAL chain, breaks replicas, and can make backups unrestorable. Fix the slot, or use pg_archivecleanup — never bare rm.

And the classic: rm -rf /var/log/journal/ while journald is running. The daemon keeps writing into file handles that now point at deleted files — you free nothing and lose your logs. Rotation tools exist because of exactly this.

The part nobody says out loud

None of these commands are hard. What's hard is running them at 2am, tired, on the right host, with nothing fat-fingered. The horror stories above aren't from people who didn't know better — they're from people who knew better and typed the wrong path anyway.

That's the gap Tryssh was built for. You tell the copilot "disk is at 89% — find what's eating it and clean up safely." It runs the read-only triage on its own — df, du, lsof +L1, the inode check — because reads can't hurt you. Then it shows you the actual cleanup, one exact command at a time, and waits for your click:

The whole triage in one ask — the reads run themselves, the cleanup waits for a click.

The investigation is automated. The rm never is. That division of labor is the whole product.

Tryssh is a native macOS SSH workspace with an agentic copilot. Download it — the first 200 agent turns are free.