Immutable deployment artifacts: stop rebuilding on the server

Treat immutable deployment artifacts: stop rebuilding on the server as a decision under uncertainty. The job is to reduce uncertainty cheaply, protect evidence, and preserve recovery options before making a change with production blast radius.

A deployment pipeline is a production system with credentials, state, concurrency, and failure modes. Fast delivery comes from small reversible changes and trustworthy evidence.

What this question is really asking

The search intent behind this problem is proving the same reviewed bytes reach every environment. That wording matters because it sets the boundary of the investigation. A vague request to “fix production” invites unrelated changes; a precise question produces a testable hypothesis and a clear finish line.

Confirm the target with hostname, environment, and service identity. Ask what changed most recently, but do not assume correlation is causation. The first goal is a narrow reproduction that another operator could repeat.

Start with evidence, not remediation

Run a compact read-only sequence:

git rev-parse HEAD
docker image inspect app-image --format '{{index .RepoDigests 0}}'
sha256sum release.tar.gz
systemctl status api

Each command should answer one question. Do not collect output simply because it looks technical. Mark what confirms normal behavior, what contradicts the working theory, and what is still unknown. If a command is unavailable, record that fact rather than silently replacing it with a riskier action.

The central interpretation for this case is: Rebuilding from mutable package indexes or branches creates an unreviewed artifact. Promote a digest-addressed result and retain it for rollback.

Build the failure chain

Describe the system as a path from the user to the dependency that completes the request. Then place each observation on that path. A strong explanation accounts for the symptom, the timing, and why healthy-looking components did not prevent the failure.

Use this sequence:

1. Reproduce the exact external symptom.
2. Bound which hosts, tenants, regions, or requests are affected.
3. Compare desired configuration with effective runtime state.
4. Read the smallest log window around the first failure.
5. Check saturation, errors, traffic, and recent changes.
6. State one hypothesis and the observation that could disprove it.
7. Choose the smallest reversible test.

If the evidence does not converge, widen one boundary at a time. Jumping from application logs to a fleet-wide restart skips the layers most likely to explain the problem.

A useful stop rule

Pause when the next action is irreversible, crosses a team boundary, exposes secrets, or lacks a tested rollback. Escalation is not failure; it is a control that prevents uncertainty from becoming damage.

Also pause when observations contradict each other. Reconfirm clocks, host identity, environment, and whether the command actually completed. Wrong-target evidence creates perfectly logical but dangerous conclusions.

Make the repair safely

Write the proposed command, expected effect, verification, and rollback before executing it. Validate syntax before reload. Keep a recovery session open during access or firewall work. For data changes, confirm backup age and restoration procedure—not merely that a backup job reported success.

After the change, test the original symptom from outside the host. Then check the adjacent failure modes: latency, errors, resource pressure, retry volume, and data correctness. “The process started” is not the same as “the service recovered.”

Turn this article into a runbook

Rehearse the sequence on a non-production target and time it. Remove commands whose output did not change a decision. The goal is not maximal coverage; it is reliable progress by a tired operator.

For the underlying model, consult GitHub Actions documentation. Product behavior and defaults change, so first-party documentation should win over copied snippets.

Related Tryssh guides

Continue with zero downtime deployment checklist, incident deploy broke production, and docker compose production. These connect the immediate symptom to access safety, incident response, and durable operating practice.

Investigate it with Tryssh

Tryssh is a native macOS SSH workspace built for this evidence-first loop. Its copilot can run safe read-only checks, retain host-specific context, and explain combined output. The visible terminal remains separate, SSH secrets stay in the macOS Keychain, and commands that change state wait for explicit approval.

That boundary is especially useful under pressure: automation gathers facts quickly, while the operator remains responsible for blast radius. Download Tryssh for macOS and rehearse the runbook on a non-production host before the next incident.