High I/O wait: find the disk queue behind the slowdown

During storage latency, the first danger is the outage. The second is the operator making an unreviewed change that expands it. A useful incident sequence is fast because it is ordered, not because it skips evidence.

Minute 0–3: verify and bound impact

Reproduce the symptom from outside the host. Identify affected regions, tenants, endpoints, and the last known-good time. Open an incident log and record every change with its timestamp. If the impact is severe, assign one person to command and one to investigate; a crowd in the shell is not redundancy.

Minute 3–8: capture high-signal evidence

Start with:

iostat -xz 1
pidstat -d 1
iotop -oPa
dmesg -T | tail -100

Capture output before restarting or redeploying. Add resource state—disk, memory, CPU, failed units—and the narrow log window around the first failure. Avoid recursive log dumps that bury the event in megabytes of unrelated history.

Minute 8–15: choose restore or diagnose

Correlate device latency and queue depth with the process issuing I/O; CPU graphs alone make this incident look mysterious.

A rollback is appropriate when a recent, reversible change aligns with the start of impact. A restart is appropriate when a known transient failure mode is understood and evidence is already captured. Neither is a substitute for an explanation, but user impact can justify restoring first and investigating from preserved data afterward.

Keep the blast radius small

• Change one layer at a time.
• Prefer reload over restart when the service supports it and state matters.
• Test configuration before applying it.
• Keep the existing SSH session open during firewall or sshd work.
• Do not delete logs, database WAL, queues, or volumes to make an alert disappear.
• Retest the original user path after every action.

Communication is part of the repair

State what users see, when the next update will arrive, and what the team is doing now. Do not announce a speculative root cause as fact. After recovery, say what changed, what remains uncertain, and which safeguards are active. Precise uncertainty builds more trust than confident guessing.

The post-incident output

A useful review produces more than a timeline. It creates a detection improvement, a safer rollback, a runbook correction, and one owner for the structural fix. If the incident depended on a single person's memory, the system is still fragile.

Use automation for evidence, not unchecked authority

Tryssh lets an agent run the read-only collection across the host while you keep control of remediation. It can correlate service status, logs, sockets, and resources in one conversation. If it proposes a restart, package change, firewall rule, or cleanup, the exact command waits for approval.

That is the right split during an incident: automation gathers quickly; a human owns the blast radius. Download Tryssh for macOS and rehearse this workflow before the next page.

Make the runbook reusable

Link the final review to a durable diagnostic guide. The journalctl guide improves evidence collection, while the website-down runbook gives the team a shared entry point for the next alert. Rehearse both during normal hours, when a mistaken assumption is cheap to correct.