Rolling Back a Deployment When Error Rate Increases

Error rate is the single most common metric behind an automatic rollback, and for good reason: it's directly observable, it's the same signal for almost every service, and a spike almost always means something. It's also the guardrail metric most often configured wrong.

What "error rate" actually needs to mean

"Error rate" isn't self-defining. Before it can back a guardrail, three things need to be decided explicitly:

• What counts as an error. HTTP 5xx only, or 4xx too? A timeout with no response at all? A retried request that eventually succeeded? Each answer changes what the metric actually measures.
• What it's a rate of. Errors per total requests (a ratio, aggregation type: average or rate depending on how it's computed upstream) behaves very differently under load than errors per second (count/sum). The first is stable across traffic volume, the second isn't.
• Over what window. A 30-second window reacts fast and false-positives on ordinary noise. A 15-minute window is stable but slow to catch a sharp regression. There's no universally correct number. It depends on how much exposure you're willing to tolerate before the guardrail can even see the problem.

Setting the threshold

The mistake teams make most often: setting the threshold at "zero errors." Every real service has a baseline error rate: retried timeouts, malformed client requests, upstream flakiness that has nothing to do with the current rollout. A threshold set at the noise floor fires constantly, and a guardrail that fires constantly gets its evaluation window widened, then its threshold loosened, then eventually gets disabled. That's the opposite of what you wanted. Set the threshold against the service's own trailing baseline, with margin, not against an idealized zero.

What automatic actually requires here

A guardrail referencing an error-rate metric definition, with a comparison operator and threshold:

POST /v1/rollouts/{rollout_id}/guardrails
{
  "metric_definition_id": "<error-rate-metric>",
  "comparison_operator": "gt",
  "threshold": 0.02,
  "evaluation_window_seconds": 300
}

When the aggregated value breaches threshold within that window, the rollout aborts through the same path a manual operator action would use. The flag in that environment goes back to serving its default variation to 100% of traffic, without the rollout's rules, and the evaluation that triggered it (the metric value, the threshold, a correlation id) is recorded, so the rollback is explainable after the fact rather than something someone has to reconstruct from memory.

The scenario this actually protects against

The regression that matters most here isn't the one visible in a dashboard someone's staring at. It's the one that happens between two check-ins, on a rollout stage nobody was watching in real time. That's the case error-rate guardrails earn their keep: not preventing every bad deploy, but converting the ones that would have run to 100% unnoticed into ones that stopped at 5% or 25%, automatically, with a record of exactly why.