Choose a Prometheus alert for duration from failure tolerance
The for clause keeps a continuously true alert pending before firing; keep_firing_for can retain firing state briefly after the expression clears.
Use Prometheus blackbox exporter to test the user-visible path
The blackbox exporter probes a target from the exporter's network and exposes probe metrics; relabeling passes the target while the exporter itself remains the scrape endpoint.
Set a Prometheus cardinality budget before adding labels
Cardinality is the number of distinct active label sets; multiplying unbounded dimensions increases memory, storage, query, and remote-write cost.
Calculate latency with histogram_quantile without averaging percentiles
Histogram_quantile estimates a quantile from histogram buckets; classic buckets must be aggregated with le preserved, while native histograms use histogram samples directly.
Use Prometheus recording rules for stable, expensive shared queries
Recording rules evaluate expressions on a schedule and store results as new series, reducing repeated dashboard cost while adding lag, storage, and naming responsibility.
Test Prometheus alerting rules with promtool before loading them
Promtool unit tests define synthetic input series and expected alerts or expressions over evaluation times, making edge cases reviewable without touching production.
Handle Prometheus missing metrics without converting absence into zero
Prometheus marks vanished series stale after scrape or discovery changes, and many expressions return no vector rather than numeric zero when a label set is absent.
Debug Prometheus target relabeling before samples disappear
Relabel_configs transforms discovered target labels before scraping, while metric_relabel_configs changes samples after scrape and before ingestion.
PromQL rate vs irate: choose by alerting and graph resolution
Rate estimates average per-second counter increase across a range and handles resets, while irate uses the last two samples and is more sensitive to short-term variation.
Build SLO burn-rate alerts from good and total events
Burn rate compares observed bad-event ratio to the error budget rate; multi-window alerts combine fast detection with evidence that impact persists.