每日查看 A/B 測試令 5% 假陽性率升至 25%，mSPRT 的代價是甚麼

Most teams I have worked with check the experiment dashboard every day. It is just what you do when a campaign is running and someone asks how it is going. The t-test on that dashboard, though, was designed for one look at a pre-agreed sample size, and the daily habit quietly breaks it.

This post measures how badly, shows the fix I used (a mixture sequential probability ratio test, mSPRT), and is explicit about what the fix costs. All numbers come from a seeded script in a public repo, linked at the end.

Why checking daily breaks the t-test

A 5% significance level means: if there is no effect and you run the test once, at the planned sample size, you will wrongly reject 5% of the time. That guarantee is about one decision.

If you look every day and stop the first time p < 0.05, you have given the random walk of your estimate many chances to wander across the line. Under no true effect, the difference between arms drifts up and down as data arrives. The chance that at least one look crosses the line grows with the number of looks, and the nominal 5% describes none of it.

Worse, the failure is silent. The p-value on the screen is computed correctly for a single look. It is the stopping rule that is wrong.

The simulation

I wanted a number, not an argument. The setup, from the repo's sequential report:

• Bernoulli outcomes, 10% base rate.
• 20 looks of 500 users per arm each, so a 10,000-per-arm horizon.
• alpha 0.05, 10,000 seeded runs per scenario.
• For the A/B scenario, the true effect is the fixed test's 80%-power minimum detectable effect (MDE), +1.19 points.

Three rules are scored on exactly the same simulated data:

The single final test lands at 4.99%, as it should. Peeking at every look with the same t-test rejects 24.8% of A/A experiments. A quarter of the experiments where nothing happened would have been reported as wins. With --large (100 looks of 100 users instead of 20 of 500, same horizon, 20,000 runs), the naive rate rises to 0.377 while mSPRT stays at 0.014.

How mSPRT works

The idea is to make the stopping rule part of the statistic, so the guarantee holds however often you look.

Start with the likelihood ratio for "the true difference is exactly delta" against "the difference is 0". You do not know delta, so instead of picking one, average the ratio over a prior on delta. I use a normal prior, delta ~ N(0, tau^2). That average is the mixture likelihood ratio, and for a difference in means it has a closed form. With d the estimated difference and V its estimated variance (s_t^2/n_t + s_c^2/n_c):

Lambda = sqrt(V / (V + tau^2)) * exp(tau^2 * d^2 / (2 V (V + tau^2)))

Two properties do the work:

1. Under "no effect", Lambda is a non-negative martingale starting at 1. Informally, its expected next value is always its current value, so it has no upward drift.
2. For such a process, Ville's inequality says the chance it ever reaches 1/alpha is at most alpha.

So the rule "stop and reject the first time Lambda >= 1/alpha" has type-I error at most alpha, however many times you inspect, whenever you stop looking, and with no correction for the number of looks.

One honest caveat: the martingale property is exact for normal data with known variance. The code plugs in sample variances, the usual large-sample approximation, so the guarantee is approximate at small n. The simulation is where I check that it holds in practice.

The code is short. This is the repo's implementation, trimmed to the arithmetic:

import numpy as np

def msprt_log_lr(estimate, variance, tau):
    tau2 = tau * tau
    d = np.asarray(estimate, dtype=np.float64)
    v = np.asarray(variance, dtype=np.float64)
    log_lr = 0.5 * np.log(v / (v + tau2)) + tau2 * d**2 / (2.0 * v * (v + tau2))
    return np.where(v > 0.0, log_lr, np.nan)

def msprt_reject(estimate, variance, tau, *, alpha=0.05):
    log_lr = msprt_log_lr(estimate, variance, tau)
    return np.nan_to_num(log_lr, nan=-np.inf) >= np.log(1.0 / alpha)

The always-valid p-value is the running minimum of 1/Lambda over the looks, so it can only go down, which is exactly what you want from something you glance at every day.

Setting the mixing scale

Validity holds for any tau > 0. The choice only moves power. The mixture concentrates its power near |delta| ~ tau, so a tau that is too small makes early stopping slow, and one that is too large spends evidence on implausibly big effects.

I set tau before the experiment to the effect the test is designed to detect: the MDE. In the simulation that is the fixed test's 80%-power MDE of +1.19 points, fixed before any data is seen. That is a design decision, not something to tune after watching the dashboard. Tuning tau on the data would quietly reintroduce the problem.

I also checked the closed form against numerical integration of the mixture in a test (test_log_lr_equals_numerical_mixture), and a second test confirms that mSPRT keeps type-I error under the null while naive peeking exceeds three times alpha.

The price

Here the table above turns uncomfortable. At the same 10,000-per-arm cap, mSPRT has power 0.465 against 0.787 for the fixed test. You cannot have both "look any time" and "same power at the same sample size".

The repo measures what it takes to get the power back (uplift-sequential-report, the price-of-peeking section):

• A larger cap. mSPRT reaches 80% power once the cap is raised to 18,000 per arm, 1.8x the fixed horizon. Its false-positive rate at that cap is 0.016, so the bigger cap costs no validity.
• More units on average, not fewer. mSPRT stops as soon as the evidence is strong, but under the alternative it still uses 11,190 per arm on average: about 12% more units than the fixed test.
• Near the cap under the null. When there is no effect it rarely stops, so it runs to 17,848 per arm on average (1.78x), close to the cap.

The trade is plain: budget for a larger maximum sample and somewhat more units on average, and in exchange you may look whenever you like without inflating false positives.

The second bullet is the one people do not expect. With this design, centred on the MDE, "sequential tests save sample" does not hold on average. I only claim this design; I have not simulated other effect sizes.

A detour: honest CUPED

The same repo covers CUPED, the other thing teams reach for to get answers faster. It replaces the outcome y with y - (X - mean(X)) @ theta, where X is a pre-experiment covariate and theta its regression coefficient. The estimate stays unbiased and the variance falls by roughly the share of outcome variance the covariate explains.

The word that matters is "predicts". On the Hillstrom e-mail RCT (64,000 customers), the pre-period covariates are past-year spend, recency, product lines, newbie and channel. They explain 2.6% of visit variance, and under 0.3% for conversion and spend. CUPED reduced the variance of the visit estimate by 2.6%. A 10% relative visit lift needs 13,217 users per arm without CUPED and 12,873 with it. The report says this plainly instead of promising a bigger number.

The method is not the problem. In the seeded simulation, where I control how well the covariate predicts the outcome, 1,000 experiments with 2,000 users per arm give:

At rho = 0.7 the variance ratio is 0.512, and the same power needs about half the users. Coverage stays near 95% throughout, so CUPED is not buying precision with validity.

When it helps: a metric with a stable history, such as weekly spend for returning customers, where last period strongly predicts this period. When it does not: the two-week visits of a one-off e-mail, where the pre-period barely says anything about who clicks. Check the correlation before you promise anyone a variance reduction.

The same habit, one level up

The uplift half of the repo has a decision problem of the same shape: four learners (S, T, X and a transformed-outcome baseline) whose test-set Qini coefficients are within noise of each other. Picking the highest point estimate is peeking in a different form: you let the data choose the winner after the fact.

So I wrote the champion rule into code (uplift/champion.py). Test the best learner against each rival with a paired bootstrap, Holm-correct the p-values, and if it is not a clear winner, take the simplest tied learner in a fixed order (S, then T, then X, then transformed outcome). On the full 13,979,592-row Criteo run with 1,000 bootstrap draws, the transformed outcome had the highest Qini point estimate, but its paired differences against S, T and X all covered 0 (Holm-adjusted p 0.35, 0.65, 0.35), so the rule registered the S-learner.

I should be precise about the timing. The rule was committed before that 1,000-draw run, but after a first full-data run had already shown the learners within noise. So it was not written blind. What it does give is a rule that is fixed in code, with its reasoning, and applied the same way to every future run, instead of a judgement call each time. The principle is the same as fixing tau in advance: decide how you will decide, ideally before the data can influence you.

Takeaways

1. If anyone looks at a running experiment, assume they will act on it. A fixed-horizon t-test checked daily produced 24.8% false positives in my simulation, and 37.7% with 100 looks.
2. If peeking is unavoidable, use a method built for it. mSPRT held 0.98% in the same setting and keeps its guarantee for any inspection schedule.
3. Plan the cost up front. At the same cap, power was 0.465 against 0.787. Getting to 80% power needed about 1.8x the cap, and the average sample under the alternative was about 12% larger than the fixed test's.
4. Fix tau at the MDE before the data arrives. It does not affect validity, but choosing it after looking would defeat the point.
5. Measure before you promise CUPED. Variance falls by about the share of outcome variance the covariate explains: 2.6% on Hillstrom visits, 49% when rho is 0.7.
6. Write down the decision rule before you see the results, whether that is a stopping rule or a model-selection rule.

The code, the seeded scripts that regenerate every number above, and the evidence table mapping each README claim to its source are at https://github.com/tantn89/uplift-experimentation.