Linux kernel log triage: hardware, filesystem, OOM, and network

Treat linux kernel log triage: hardware, filesystem, oom, and network as a decision under uncertainty. The job is to reduce uncertainty cheaply, protect evidence, and preserve recovery options before making a change with production blast radius.

Linux performance is a queueing problem expressed through CPU, memory, storage, network, and scheduler behavior. Start with pressure and latency, then identify the responsible workload.

What this question is really asking

The search intent behind this problem is separating actionable new events from old boot noise. That wording matters because it sets the boundary of the investigation. A vague request to “fix production” invites unrelated changes; a precise question produces a testable hypothesis and a clear finish line.

Confirm the target with hostname, environment, and service identity. Ask what changed most recently, but do not assume correlation is causation. The first goal is a narrow reproduction that another operator could repeat.

Start with evidence, not remediation

Run a compact read-only sequence:

journalctl -k --since '1 hour ago'
dmesg --level=err,warn
uptime -s
last -x | head

Each command should answer one question. Do not collect output simply because it looks technical. Mark what confirms normal behavior, what contradicts the working theory, and what is still unknown. If a command is unavailable, record that fact rather than silently replacing it with a riskier action.

The central interpretation for this case is: Correlate monotonic and wall-clock time with the symptom. Repeated device resets, filesystem errors, OOM kills, and link changes need different owners and urgency.

Build the failure chain

Describe the system as a path from the user to the dependency that completes the request. Then place each observation on that path. A strong explanation accounts for the symptom, the timing, and why healthy-looking components did not prevent the failure.

Use this sequence:

1. Reproduce the exact external symptom.
2. Bound which hosts, tenants, regions, or requests are affected.
3. Compare desired configuration with effective runtime state.
4. Read the smallest log window around the first failure.
5. Check saturation, errors, traffic, and recent changes.
6. State one hypothesis and the observation that could disprove it.
7. Choose the smallest reversible test.

If the evidence does not converge, widen one boundary at a time. Jumping from application logs to a fleet-wide restart skips the layers most likely to explain the problem.

A useful stop rule

Pause when the next action is irreversible, crosses a team boundary, exposes secrets, or lacks a tested rollback. Escalation is not failure; it is a control that prevents uncertainty from becoming damage.

Also pause when observations contradict each other. Reconfirm clocks, host identity, environment, and whether the command actually completed. Wrong-target evidence creates perfectly logical but dangerous conclusions.

Make the repair safely

Write the proposed command, expected effect, verification, and rollback before executing it. Validate syntax before reload. Keep a recovery session open during access or firewall work. For data changes, confirm backup age and restoration procedure—not merely that a backup job reported success.

After the change, test the original symptom from outside the host. Then check the adjacent failure modes: latency, errors, resource pressure, retry volume, and data correctness. “The process started” is not the same as “the service recovered.”

Turn this article into a runbook

Rehearse the sequence on a non-production target and time it. Remove commands whose output did not change a decision. The goal is not maximal coverage; it is reliable progress by a tired operator.

For the underlying model, consult Linux kernel administration documentation. Product behavior and defaults change, so first-party documentation should win over copied snippets.

Related Tryssh guides

Continue with linux performance first 10 minutes, linux high cpu, and linux high memory. These connect the immediate symptom to access safety, incident response, and durable operating practice.

Investigate it with Tryssh

Tryssh is a native macOS SSH workspace built for this evidence-first loop. Its copilot can run safe read-only checks, retain host-specific context, and explain combined output. The visible terminal remains separate, SSH secrets stay in the macOS Keychain, and commands that change state wait for explicit approval.

That boundary is especially useful under pressure: automation gathers facts quickly, while the operator remains responsible for blast radius. Download Tryssh for macOS and rehearse the runbook on a non-production host before the next incident.