logrotate postrotate failure: diagnosis and repair
logrotate postrotate failure is an incident description, not a root cause. Start by proving the host, affected path, time window, and first broken boundary.
TL;DR: postrotate failure is a resource pressure incident in logrotate. Capture one synchronized evidence bundle, compare the four operational conditions below, make one reversible change, and verify the original user path.
Audience: operators comfortable with SSH, service logs, and basic Linux administration. Keep a provider console, second session, replica, or other independent recovery path before changing production.
What logrotate postrotate failure means
The visible symptom can be the initiating fault, a protection mechanism, or downstream damage. Treat it as a case file: preserve order, distinguish causes from consequences, and prefer a falsifiable mechanism over a familiar error label.
Open the incident like a case file
Record the exact hostname, role, environment, UTC time, first affected request or resource, last known success, affected peers, recent changes, and an independent recovery path. A remote shell makes it easy to investigate the wrong replica with complete confidence.
A mental model for the failure path
Read the path from left to right:
- Entry: a request, scheduled event, packet, write, or control-plane instruction arrives.
- Admission: identity, policy, capacity, and configuration decide whether work may enter.
- Execution: logrotate consumes CPU, memory, descriptors, storage, locks, or downstream calls.
- Commit: state becomes durable or visible to the next component.
- Observation: logs, metrics, health checks, and client errors report different points on that path.
The first contradictory transition is more valuable than the loudest log line. A successful health endpoint proves only the code behind that endpoint. It does not prove that a real request can traverse DNS, TLS, admission, execution, and persistence.
Collect a synchronized evidence bundle
Run one line at a time and read it before execution:
# Component-specific state
logrotate --debug /etc/logrotate.conf; systemctl status logrotate.timer --no-pager; journalctl -u logrotate -n 120 --no-pager; df -h
# Category-specific host pressure or policy
free -h; df -hT; df -i; cat /proc/pressure/memory; ss -s
# Logs bounded to the incident window
journalctl -u logrotate --since '30 minutes ago' --no-pager | tail -n 120Capture the exit status and first decisive line. “No output” is evidence only when the command, permissions, time range, and log destination are known. Redact credentials, customer data, private addresses, and complete environment dumps.
Compare the four operational conditions
The condition is useful only when it changes the mechanism, evidence, or disconfirming test.
After reboot
The failure begins only after a boot boundary, so persisted configuration, mount ordering, generated state, and service dependencies matter more than the current shell.
Disconfirming test: If the same signature reproduces without rebooting, boot ordering is probably a coincidence rather than the cause.
Under load
The quiet system looks healthy, but concurrency exposes a finite queue, scheduler delay, memory pressure, timeout budget, or downstream capacity limit.
Disconfirming test: If the failure rate does not move with concurrency or pressure, look for time, identity, or configuration boundaries instead.
After an upgrade
A version, dependency, default, file format, feature gate, or generated unit changed; temporal correlation is useful evidence but is not yet causation.
Disconfirming test: If old and new nodes show the same signature under the same traffic, the upgrade may only have revealed an older capacity problem.
Intermittently
The incident crosses a timing, topology, cache, leader, retry, or workload boundary, so one successful check cannot disprove the failure.
Disconfirming test: If every request through one exact path fails, replace probabilistic sampling with a deterministic path comparison.
Rank hypotheses before touching production
- State the leading mechanism, not an error label.
- Name supporting observations from at least two layers.
- Write one test that would make the explanation wrong.
- Keep the strongest competing explanation visible.
- Define the smallest repair boundary.
- Write the rollback before the change.
Avoid explanations such as “the upgrade caused it,” “the network is flaky,” or “the server is overloaded.” They describe timing or sensation, not a mechanism that predicts the next observation.
Repair without widening the incident
The repair objective is to name the exhausted resource and owner, stop unbounded growth, then change one measured limit with headroom and alerts. Prefer a change that can be observed independently and reverted without the path being edited.
Before a service reload, validate syntax with the component-supported checker. Before changing storage or replicated state, protect the best surviving copy. Before changing firewall, routing, identity, or mandatory-access policy, keep an out-of-band session open. Before raising capacity, establish why consumption grew and where the next ceiling sits.
The category-specific warning is: Raising every limit transfers the failure to the next layer and can turn a controlled rejection into host-wide collapse.
Make one change, record its UTC time, and wait for the system's real convergence boundary. Reload completion, process health, replica catch-up, cache expiry, and client recovery are not the same event.
Verify from the outside in
$ date -u; hostname -f Repeat the original user action, not only a process health check.
A repair is complete when a fresh request through the original path succeeds, the causal signal recovers, adjacent controls still work, and the change remains stable across the relevant convergence boundary.
Why tempting fixes fail
Restart everything
A restart may release memory, locks, sockets, or stale state, but it also destroys ordering evidence. If restart is required to restore service, capture bounded logs, process state, pressure, versions, and configuration paths first, then label the action as mitigation rather than root-cause repair.
Increase every limit
Higher limits can postpone rejection while increasing blast radius. Measure the constrained resource, its owner, arrival rate, service rate, and downstream ceiling. Capacity without backpressure is not resilience.
Disable the control that reports the problem
Turning off certificate verification, SELinux, AppArmor, authentication, health checks, or integrity checks replaces a visible failure with unobserved risk. Prove the exact denied operation and repair its trust relationship.
Copy configuration from a healthy server
Peers often have different roles, secrets, disks, addresses, feature gates, or generated state. Compare effective settings and explain each difference. Never overwrite the only evidence of drift.
Evidence and review status
This guide consolidates four condition-targeted drafts into one canonical incident resource. Commands and technical claims are linked to upstream documentation. The page does not claim that the failure was reproduced on every component or version; measured examples must identify the host, version, method, and captured artifact.
Limits of Tryssh
Tryssh is an SSH workspace and agentic copilot, not a monitoring system, cluster controller, backup product, or automatic incident commander. It sees only authorized hosts, commands, and outputs. Network partitions, compromised hosts, incomplete logs, and missing telemetry leave genuine uncertainty. State-changing commands remain human decisions.
Related server runbooks
Continue with logrotate rotation state lock, logrotate inode retention, health checks versus monitoring, and the SSH hardening checklist.