A/B 測試實際需要多少訪客？附樣本量表格及免費計算器

Most A/B tests that "won" never had enough visitors to win anything. Here is the math, a table you can bookmark, and a tiny zero-dependency calculator.

The short answer

At a 2% conversion rate, detecting a 20% relative lift (2.0% to 2.4%) takes about 21,106 visitors per version at 95% confidence and 80% power. Two versions means about 42,000 visitors on that page.

Smaller lifts and lower conversion rates need a lot more:

Visitors per version. Two-sided test, 95% confidence, 80% power.

The formula

For two proportions p1 (current) and p2 (with the lift):

n = (z_a/2 + z_b)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2

with z_a/2 = 1.96 for 95% confidence (two-sided) and z_b = 0.84 for 80% power.

In JavaScript:

const Z = 1.959964 + 0.841621;
function sampleSize(base, lift) {
  const p2 = base * (1 + lift);
  return Math.ceil((Z * Z * (base * (1 - base) + p2 * (1 - p2))) / (p2 - base) ** 2);
}
sampleSize(0.02, 0.2); // 21106

I put this (plus an inverse-normal function so you can change confidence and power, and a CLI) in a small MIT repo: baneets/ab-test-sample-size.

npx github:baneets/ab-test-sample-size 2 20 50000
# 21,106 visitors per version (42,212 total), 95% confidence, 80% power
# about 3.7 weeks at 50,000 visitors a month

How long that takes

A page with 50,000 visitors a month at 2% needs about 3.7 weeks to detect a 20% lift, and about 14 weeks for a 10% lift. At 10,000 visitors a month, the 20% case takes about 18 weeks.

That is why button-colour tests on low-traffic stores almost never produce a real answer.

Three mistakes that create fake winners

1. Peeking. Checking a fixed-sample test every day and stopping the first time it looks significant inflates false positives a lot. Either commit to the sample size, or use a method built to be checked as it runs (sequential tests, Bayesian tests, multi-armed bandits).
2. Tiny effects on small traffic. If your page gets 5,000 visits a month, test big changes (offer, headline, first screen), not small ones. Big lifts need far fewer visitors.
3. Not running whole weeks. People buy differently on Monday and Saturday. Run at least one full week even if the number is reached sooner.

Bandits vs fixed tests

A classic A/B test splits traffic 50/50 until the sample is reached. A multi-armed bandit (often Thompson sampling) moves traffic toward the version that is winning while the test runs, so fewer visitors see the loser. The catch: a pure bandit can starve your original page, which makes the gain hard to prove. The fix is to keep a holdout share on the original no matter what.

I wrote more on that here: multi-armed bandit vs A/B testing.

Try it in the browser

If you just want the number without installing anything, there is an interactive version here: A/B test sample size calculator.

Disclosure: I build Luria, an AI split-testing tool, which is why I care about this. The math above is standard and works with any testing tool.