A/B test significance calculator
A difference between two conversion rates is only meaningful if it is bigger than the noise. This compares the two.
Confidence
95.256%
- Two-tailed p-value
- 0.047 438
- Z-score
- 1.9824
- Conversion rate A
- 0.050 00
- Conversion rate B
- 0.059 00
- Relative lift
- 18.000 %
- Significant at 5%
- 1
Show the working
- Visitors, variant A = 5000
- Conversions, variant A = 250
- Visitors, variant B = 5000
- Conversions, variant B = 295
- Conversion rate A = 250 / 5000 = 0.05
- Conversion rate B = 295 / 5000 = 0.059
- Relative lift = 0.05 > 0 ? (0.059 - 0.05)/0.05*100 : 0 = 18
- pooled = (250 + 295) / (5000 + 5000) = 0.0545
- se = sqrt(0.0545 * (1 - 0.0545) * (1/5000 + 1/5000)) = 0.004 540 03
- Z-score = 0.004 540 03 > 0 ? (0.059 - 0.05) / 0.004 540 03 : 0 = 1.982 36
- z_abs = abs(1.982 36) = 1.982 36
- t = 1 / (1 + 0.2316419 * 1.982 36) = 0.685 308
- pdf = 0.3989422804014327 * exp(-1.982 36^2 / 2) = 0.055 920 6
- tail = 0.055 920 6 * (0.319381530*0.685 308 - 0.356563782*0.685 308^2 + 1.781477937*0.685 308^3 - 1.821255978*0.685 308^4 + 1.330274429*0.685 308^5) = 0.023 719 2
- Two-tailed p-value = 2 * 0.023 719 2 = 0.047 438 3
- Confidence = (1 - 0.047 438 3) * 100 = 95.2562
- Significant at 5% = 0.047 438 3 < 0.05 ? 1 : 0 = 1
Values are shown to six significant figures here; the arithmetic itself runs on exact fractions and is only rounded where a number is printed.
Scenarios & local history
Compare a saved scenario
Recent calculations
The formula
- pooled rate p = (cA + cB) / (nA + nB)
- standard error = sqrt(p(1-p)(1/nA + 1/nB))
- z = (rateB - rateA) / standard error
Stopping a test as soon as it crosses significance inflates the false positive rate badly. Fix the sample size in advance.
Statistical significance is not business significance. A real but tiny lift may not be worth shipping.