Hypothetical example: this scenario is invented to show the method. It does not describe a rollfuse customer, an internal measurement or a real incident.
A rollout guardrail set to abort on p95 latency will do exactly what it's told: watch the median-ish middle of the distribution, and stay quiet as long as that middle stays healthy. On a payment gateway with multiple issuers, retry logic, and anti-fraud checks in the path, the tail is where things actually break, and p95 doesn't see the tail.
The shape of the failure
HYPOTHETICAL TIMELINE
The guardrail wasn't misconfigured in any obvious way. It was calibrated, tested, and had a track record of firing correctly on previous rollouts. It just wasn't watching the metric where this particular class of regression shows up. p95 stayed inside its threshold for the entire window.
Why p95 alone isn't enough here
Most teams default their latency guardrails to p95 because it's the standard choice: it captures broad degradation without reacting to isolated outliers. That default is reasonable for a service with a homogeneous dependency graph. It's insufficient for a service like a payment gateway, where the tail specifically, retries against a slow issuer, an antifraud check that occasionally times out, can represent a disproportionately consequential slice of traffic, even at a small percentage.
THE FIX
A rollout on a service with a heterogeneous external dependency should carry two latency guardrails, not one: one referencing a p95 metric definition, one referencing p99. rollfuse does not compute percentiles: each guardrail compares the average of the observations in its window against the threshold, so the p95 and p99 values have to arrive already calculated by your metric source. Either guardrail breaching aborts the rollout independently, so neither metric has to "agree" with the other before the rollout stops.
Why manual detection got there first
In the timeline above, seven minutes separate the p99 breach from the first alert, and that alert came from a person, not the system. That's the opposite of what a guardrail is supposed to deliver: the entire point is catching a regression before a customer notices, not after. It's worth auditing your own guardrails specifically for this gap: any latency guardrail configured for p95 only, on a service where the tail matters, is carrying the same blind spot.