Docker Compose outage: dependency order is not readiness
During a multi-container stack that starts but does not work, the first danger is the outage. The second is the operator making an unreviewed change that expands it. A useful incident sequence is fast because it is ordered, not because it skips evidence.
Minute 0–3: verify and bound impact
Reproduce the symptom from outside the host. Identify affected regions, tenants, endpoints, and the last known-good time. Open an incident log and record every change with its timestamp. If the impact is severe, assign one person to command and one to investigate; a crowd in the shell is not redundancy.
Minute 3–8: capture high-signal evidence
Start with:
docker compose ps
docker compose logs --since 20m
docker inspect --format '{{json .State.Health}}' service
Capture output before restarting or redeploying. Add resource state—disk, memory, CPU, failed units—and the narrow log window around the first failure. Avoid recursive log dumps that bury the event in megabytes of unrelated history.
Minute 8–15: choose restore or diagnose
Look for health, DNS, mounts, and dependency readiness rather than assuming container running means application ready.
A rollback is appropriate when a recent, reversible change aligns with the start of impact. A restart is appropriate when a known transient failure mode is understood and evidence is already captured. Neither is a substitute for an explanation, but user impact can justify restoring first and investigating from preserved data afterward.
Keep the blast radius small
- Change one layer at a time.
- Prefer reload over restart when the service supports it and state matters.
- Test configuration before applying it.
- Keep the existing SSH session open during firewall or sshd work.
- Do not delete logs, database WAL, queues, or volumes to make an alert disappear.
- Retest the original user path after every action.
Communication is part of the repair
State what users see, when the next update will arrive, and what the team is doing now. Do not announce a speculative root cause as fact. After recovery, say what changed, what remains uncertain, and which safeguards are active. Precise uncertainty builds more trust than confident guessing.
The post-incident output
A useful review produces more than a timeline. It creates a detection improvement, a safer rollback, a runbook correction, and one owner for the structural fix. If the incident depended on a single person's memory, the system is still fragile.
Use automation for evidence, not unchecked authority
Tryssh lets an agent run the read-only collection across the host while you keep control of remediation. It can correlate service status, logs, sockets, and resources in one conversation. If it proposes a restart, package change, firewall rule, or cleanup, the exact command waits for approval.
That is the right split during an incident: automation gathers quickly; a human owns the blast radius. Download Tryssh for macOS and rehearse this workflow before the next page.
Make the runbook reusable
Link the final review to a durable diagnostic guide. The journalctl guide improves evidence collection, while the website-down runbook gives the team a shared entry point for the next alert. Rehearse both during normal hours, when a mistaken assumption is cheap to correct.