Reading Experiment Results and Deciding When to Conclude

A running experiment accumulates two kinds of data: exposures (who saw which variation) and observations (who converted on the metric you're tracking). Two endpoints turn that raw data into a decision: sample-ratio-mismatch, which tells you whether the exposure data itself can be trusted, and outcome-analysis, which tells you whether one variation actually beat the other. You check the first before you believe the second.

Check for a sample ratio mismatch first

const res = await fetch(
  `https://api.rollfuse.com/v1/experiments/${experimentId}/sample-ratio-mismatch`,
  { headers: { Authorization: `Bearer ${token}` } },
);

const srm = await res.json();

An experiment with a 50/50 allocation should end up with roughly equal exposure counts in each variation. sample-ratio-mismatch runs a chi-squared test comparing the exposure counts you actually got against the allocation you configured, and flags a mismatch when the deviation is statistically unlikely to be chance.

A mismatch almost never means "the treatment variation is unpopular." It means something is wrong with how subjects got bucketed or counted, for example:

• A bug in the client that causes one variation's exposure event to be dropped more often than the other's (a slow render path, a crash before the exposure fires).
• A cache or CDN serving a stale Configuration to part of your traffic, so some subjects never see the experiment's bucketing rules at all.
• Bot or retry traffic that hits one code path more than the other and inflates its exposure count.

If sample-ratio-mismatch reports a mismatch, do not read the outcome analysis as ground truth. The two variations were not tested on comparable populations, so any lift the outcome analysis reports could be explained entirely by whichever bug is causing the mismatch rather than the actual product change. Fix the underlying cause and, if enough of the experiment's traffic was affected, restart it rather than trying to correct for the skew statistically.

Read the outcome analysis

Once exposure counts look sound, outcome-analysis gives you the comparison between variations on the metric you've been recording with observations.

const res = await fetch(
  `https://api.rollfuse.com/v1/experiments/${experimentId}/outcome-analysis`,
  { headers: { Authorization: `Bearer ${token}` } },
);

const outcome = await res.json();

for (const variation of outcome.variations) {
  console.log(
    variation.key,
    "conversion:", variation.conversion_rate,
    "lift vs control:", variation.relative_lift,
    "p-value:", variation.p_value,
  );
}

Three numbers matter for the decision:

• Conversion rate per variation: observations divided by exposures for that variation. This is the raw signal before any statistical adjustment.
• Relative lift: how much better (or worse) a treatment variation's conversion rate is compared to control, as a percentage. A positive lift with a low p-value is what you're looking for when you expected the treatment to win.
• p-value / significance: the probability that a lift at least this large would happen by chance if the two variations were actually identical. rollfuse flags a variation as statistically significant once its p-value crosses below the experiment's configured significance threshold (0.05 by default). A low sample size can produce a large-looking lift with a high p-value; that lift is noise, not a result, and should not be acted on yet.

Do not stop an experiment the moment one variation pulls ahead in raw conversion rate. Early in an experiment, small differences in exposure counts produce large, unstable swings in conversion rate that regress toward the actual effect as more data comes in. The p-value is what tells you whether you've collected enough evidence to trust the direction you're seeing, not just its magnitude.

Conclude or abort based on what you found

// the treatment variation is significantly better: end the experiment and keep it
await fetch(
  `https://api.rollfuse.com/v1/experiments/${experimentId}/conclude`,
  { method: "POST", headers: { Authorization: `Bearer ${token}` } },
);

// sample-ratio-mismatch flagged a problem, or the treatment is clearly worse: stop early
await fetch(
  `https://api.rollfuse.com/v1/experiments/${experimentId}/abort`,
  { method: "POST", headers: { Authorization: `Bearer ${token}` } },
);

Use conclude when the experiment has run long enough to reach significance on its primary metric (or has clearly failed to move it after collecting enough exposures to have the statistical power to detect a real effect) and the exposure data passed the sample-ratio-mismatch check. Concluding locks in the result and is the correct outcome whether the treatment won, lost, or tied.

Use abort when something invalidates the experiment before you can trust its result at all: a confirmed sample ratio mismatch you can't fix quickly, a treatment variation causing a regression severe enough that you're not willing to wait for statistical significance to prove it, or a rollout that needs to be reverted for unrelated operational reasons. Aborting stops the experiment without asserting a winner, which is the honest state for a run whose data you don't trust.