← All posts Deep dive

The API is 500ing and it's 3am. A field guide.

The Tryssh team · July 20, 2026

There's a specific kind of thinking you do at 3am, and it isn't good. You're sleep-deprived, the pager is still buzzing, and you're one mistyped path away from making a bad night worse. Which is exactly why incident response shouldn't run on memory and adrenaline — it should run on a sequence.

Here's the one that works, and the four root causes it almost always finds.

First, establish when — not why

The single most valuable fact in any outage is the timestamp of the first error. Everything correlates from there: the deploy that landed five minutes before, the cron job that fires at that hour, the cert that expired at exactly midnight.

curl -v https://api.example.com/health
journalctl -u your-app --since "02:30" -p err --no-pager
journalctl -u nginx -n 50 --no-pager
git log --since="6 hours ago" --oneline    # did anyone deploy?

Then a resource snapshot, because half of all mystery outages are just a machine running out of something:

df -h && df -i        # space AND inodes — they fail with the same error
free -h
ps aux --sort=-%mem | head
ss -tan | awk '{print $1}' | sort | uniq -c

Five commands, and you've usually got a suspect. The archetypes below cover most nights.

Archetype 1: The service "shut down cleanly" — it didn't

The process exited at 02:41 with no stack trace, no core dump, nothing in the app log. It looks like a clean shutdown. It wasn't — the kernel killed it.

journalctl -b | grep -i oom

If you see oom-kill: task node[1234] score 891, the OOM killer took your process, and the app log is silent because the app never got a chance to say anything. The real question becomes what ate the memory — often a slow query ballooning a connection pool, or a leak that took three weeks to matter.

Archetype 2: Everything is "slow," nothing is "down"

Response times climb from 50ms to 5 seconds to timeout. The database CPU is at 5%. Nothing is obviously broken, which makes it the most confusing failure mode there is.

This is connection pool exhaustion. Some endpoint is checking out connections and never returning them; after the fiftieth request, everyone else queues.

ss -tan | grep -c ESTAB
psql -c "SELECT state, count(*) FROM pg_stat_activity GROUP BY state;"

A pile of connections in idle in transaction is your answer. The tell that separates it from a database problem: the DB is bored while the app starves.

Archetype 3: It broke at exactly midnight

When a failure has a suspiciously round timestamp, it's a certificate. Renewal failed silently weeks ago — a DNS record changed, a challenge broke — and nobody reads certbot's logs until the cert actually lapses.

echo | openssl s_client -connect api.example.com:443 2>/dev/null | openssl x509 -noout -dates
tail -50 /var/log/letsencrypt/letsencrypt.log

Thirty seconds to confirm, and it explains why "the API is down" while every process on the box is running fine.

Archetype 4: "No space left on device" — but df shows space

Writes fail, the error says the disk is full, df -h says 50% free. Check inodes: df -i. A queue directory quietly accumulated two million tiny files, and the filesystem ran out of inodes long before it ran out of blocks. (We wrote a whole piece on disk triage — read it here.)

The mistakes that make it worse

Every one of these is a thing a competent engineer did while tired:

  • Restarting the service before understanding why it died. Sometimes it works. Sometimes it kills the evidence and cascades — the restart storms a dependency that was barely holding on.
  • rm-ing a log file the process still holds open. Frees nothing (the handle keeps the blocks alive), destroys the evidence, and the disk fills again in two hours.
  • kill -9 by name. pkill -9 python takes out the app, the backup job, and the monitoring agent in one keystroke.
  • Editing a config "just to test" and reloading — now there are two problems, and tomorrow-you won't remember the edit.

Notice the pattern: the reads were never the danger. journalctl, df, ss, openssl — you can run those all night without risk. Every 3am disaster comes from a mutation made in a hurry.

Structure beats willpower

That read/write asymmetry is the entire design of Tryssh. You open the laptop, type "api is 500ing since 02:40 — find out why", and the copilot runs the whole diagnostic sequence above over SSH — logs, resources, connections, cert dates — because read-only commands auto-run through its risk gate. It comes back with the root cause and a proposed fix:

The diagnosis runs itself; the restart waits for you.

The fix — the restart, the rollback, the config change — renders as an exact command and waits for your click. At 3am, that's not a formality. That's the thing standing between you and the war stories above.

Tryssh is a native macOS SSH workspace with an agentic copilot. Download it free — 200 agent turns included.