← All posts Prometheus

Calculate latency with histogram_quantile without averaging percentiles

The Tryssh team ·

For Prometheus histogram_quantile, the safest approach is a bounded operational change, not a command pasted without context. This runbook starts with effective state, shows the smallest candidate action, and finishes by repeating the real user or system path.

TL;DR: Histogram_quantile estimates a quantile from histogram buckets; classic buckets must be aggregated with le preserved, while native histograms use histogram samples directly. Capture a baseline with promtool query instant http://127.0.0.1:9090 'sum by (le) (rate(http_request_duration_seconds_bucket[5m]))', make the reviewed change only when the evidence matches, then verify with promtool query instant http://127.0.0.1:9090 'sum(rate(http_request_duration_seconds_count[5m]))' and keep the rollback ready.

Audience: operators who can inspect Prometheus targets, labels, rules, and query results without treating a dashboard as ground truth. This guide assumes familiarity with Prometheus operations and a change window appropriate to the system.

Calculate latency with histogram_quantile without averaging percentiles observe, interpret, change, and verify workflow

The direct answer

The quantile is supported by useful bucket boundaries and enough observations for the selected interval. That is the success condition for Prometheus histogram_quantile; command completion by itself is not enough.

The important boundary is instrumentation, scrape discovery, relabeling, sample ingestion, query evaluation, rule state, and notification delivery. Histogram_quantile estimates a quantile from histogram buckets; classic buckets must be aggregated with le preserved, while native histograms use histogram samples directly. If an observation does not identify which side of that boundary failed, collect a narrower observation before changing state.

How the mechanism works

For Prometheus histogram_quantile, use this mental model: Prometheus discovers targets, transforms labels, scrapes timestamped samples, evaluates PromQL over those series, and turns rule results into recorded series or alert states. The model prevents a common mistake—treating configuration text, control-plane acceptance, process state, and end-user behavior as the same proof.

Follow four stages:

  1. Observe: identify the exact host, object, version, owner, and active configuration.
  2. Interpret: write the expected result, the abnormal result, and what would remain inconclusive.
  3. Change: apply one reviewed action at the narrowest layer that contradicts the baseline.
  4. Verify: repeat the original path and compare the same evidence, including adjacent safety controls.

Preflight and safety boundary

Run new queries over bounded ranges, estimate series growth, test rule files, and keep existing alerts until replacement coverage is proven.

Before Prometheus histogram_quantile, record UTC time, the current version or digest, the exact target, recent changes, and who owns the workload. The rollback for this runbook is: restore the prior dashboard or rule, then redesign buckets or adopt supported native histograms before changing objectives.

Do not continue if the target identity is ambiguous, the current state cannot be saved, the only recovery session would be at risk, or the proposed command affects more objects than the brief names.

Capture the read-only baseline

Run these commands one at a time. Replace example names and addresses deliberately; do not paste production secrets into a transcript.

promtool query instant http://127.0.0.1:9090 'sum by (le) (rate(http_request_duration_seconds_bucket[5m]))'
promtool query instant http://127.0.0.1:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))'

Interpret the baseline before moving on:

  • Expected: the quantile is supported by useful bucket boundaries and enough observations for the selected interval.
  • Abnormal: the result sits inside a wide bucket, traffic is sparse, or incompatible buckets are aggregated.
  • Inconclusive: missing output can also mean the wrong context, permissions, namespace, log window, binary, or target. Prove those assumptions before treating absence as health.

Save the decisive output, exit status, and timestamp. Redact credentials, customer data, private topology, tokens, and complete environment dumps.

Apply the smallest candidate change

The following is state-changing example syntax, not an instruction to run it unchanged:

promtool query range --start='2026-07-29T00:00:00Z' --end='2026-07-29T01:00:00Z' --step=30s http://127.0.0.1:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))'

For Prometheus histogram_quantile, the proposed change is acceptable only when the read-only baseline predicts its effect and the rollback is available. The key risk is: averaging per-instance p95 values produces a number that is not the fleet p95.

Prefer an immutable artifact, validated configuration, dry-run, transaction, candidate object, or staged target when the tool supports one. Record the exact command and UTC time so later telemetry can be correlated to the change.

Verify the result from the outside in

promtool query instant http://127.0.0.1:9090 'sum(rate(http_request_duration_seconds_count[5m]))'
promtool query instant http://127.0.0.1:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))'

Verification for Prometheus histogram_quantile has three layers:

  1. The control plane or command reports the intended effective state.
  2. The process, resource, or data path reflects that state without a new pressure signal.
  3. The original user-visible or dependent-system path succeeds from an independent vantage point.

If promtool query instant http://127.0.0.1:9090 'sum(rate(http_request_duration_seconds_count[5m]))' succeeds but the original path still fails, stop. The change may have repaired a local symptom while DNS, policy, routing, caching, dependency, or client state remains broken.

Failure branches

The baseline does not match this runbook

When the result sits inside a wide bucket, traffic is sparse, or incompatible buckets are aggregated, do not force the candidate command. Return to identity and scope, compare a healthy peer only through effective settings, and name a new falsifiable mechanism.

The change succeeds but behavior does not

A successful promtool query range --start='2026-07-29T00:00:00Z' --end='2026-07-29T01:00:00Z' --step=30s http://127.0.0.1:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))' proves that one interface accepted a request. It does not prove convergence, readiness, data compatibility, external routing, or client recovery. Re-run the same evidence at each downstream boundary.

The change makes the system worse

Execute the written rollback: restore the prior dashboard or rule, then redesign buckets or adopt supported native histograms before changing objectives. Preserve the failed candidate and relevant logs long enough to explain the outcome; do not destroy the evidence with broad cleanup or repeated restarts.

Operator checklist

  • Confirm the exact target, context, identity, version, and active owner.
  • Capture the read-only baseline and one disconfirming observation.
  • Label promtool query range --start='2026-07-29T00:00:00Z' --end='2026-07-29T01:00:00Z' --step=30s http://127.0.0.1:9090 'histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))' as state-changing during review.
  • Keep recovery access and rollback independent of the path being edited.
  • Change one layer, record UTC time, and wait for its real convergence boundary.
  • Verify the original path, adjacent controls, resource pressure, and persistence.
  • Update the runbook when observed behavior differs from the source-reviewed model.

Investigate it in Tryssh

Tryssh keeps the command, approval boundary, output, and verification beside the host conversation.

Tryssh can preserve this evidence loop and show a state-changing command for human approval. It does not make the operator's identity, recovery access, rollback, or platform authority decisions.

Evidence and review status

This Prometheus histogram_quantile runbook was source-reviewed on 2026-07-29 against current first-party documentation. The commands are illustrative and use example targets. The page does not claim that the change was reproduced across every distribution, managed service, version, network, or workload.

Limitations and trade-offs

Averaging per-instance p95 values produces a number that is not the fleet p95. Managed platforms may generate configuration, restrict privileges, replace local state, or expose a different control plane than the upstream project. Confirm the installed version and provider contract before applying a repair.

Search visibility is not proof of operational correctness. Treat this page as a decision aid, preserve independent recovery, and stop when the evidence contradicts its assumptions.

Related operator runbooks

Continue with Prometheus recording rules, PromQL rate vs irate, the Prometheus operations foundation guide, and the SSH hardening checklist.

Sources and further reading