You changed the prompt. Nobody knows what that did to the answers.
Swapping a model or editing a prompt changes behavior you cannot enumerate in tests. Treat it as a rollout: a share of traffic first, quality and cost on guardrails, and a record of why the new version won.
Creating an account with Google shares your name, email address and profile picture with us, so we can identify you and contact you about your account. Our stated retention period is the life of the account plus 30 days after you close it.
Your test suite cannot tell you the answers got worse
The build passes. The types check. The output is different in a way no assertion in your repository was written to catch.
The regression is qualitative
A prompt edit that fixes one class of answer quietly degrades another. You find out from a support ticket, weeks later, phrased as the product feeling worse.
Cost and latency move independently of quality
The new version answers better and costs three times as much per request. Neither the eval suite nor the pull request mentions that, because neither was watching.
Nobody can reconstruct why the version shipped
Someone decided the new one was better. The reasoning was in a thread, the numbers were in a notebook, and the notebook has been re-run since.
What changes when a model version is a rollout
The same version change, exposed on terms you set, with a decision you can reproduce.
A share of requests, not all of them
Deterministic targeting keeps the same user on the same version for the whole rollout, so a comparison is not polluted by people flipping between versions mid-session.
Quality, cost and latency all get a guardrail
Any one of them crossing its threshold pauses the promotion. A version that answers better but costs three times more stops on its own.
The promotion carries its reason
The metric values behind each stage, and the automated action taken, are recorded with the rollout. The answer to why this version shipped is a record, not a memory.
How it works
Make the model version a flag variation
Prompt, model and parameters become configuration, so trying a version does not require shipping code.
Send your own quality signal as a metric
Thumbs-up rate, escalation rate, whatever you already collect. It becomes a first-class metric the guardrail can act on.
Let promotion depend on all three
Quality, cost and latency each get a threshold. The rollout only advances when none of them is breached.
Promote your next model version on evidence
Fifteen minutes from signup to a model variation serving a share of real traffic behind guardrails.
Start on GrowthCreating an account with Google shares your name, email address and profile picture with us, so we can identify you and contact you about your account. Our stated retention period is the life of the account plus 30 days after you close it.
Unlimited experiments, guardrail auto-rollback and reusable metrics included on Growth.