Tools
A/B test significance calculator
Enter visitors and conversions for your control and variant. Get the verdict instantly — p-value, z-score, confidence interval, observed power, an automatic sample-ratio check, and the distributions behind the numbers.
Test setup
Control (A)
Variant (B)
Runs entirely in your browser — nothing you enter is stored or sent anywhere.
Significant at 95% confidence
Variant B beats control A
13.00% vs 10.00% conversion. You can be 95% confident this is not due to chance.
SRM balanced (50.00% on A)
+30.00%
relative uplift
P-value (two-sided)
0.0355
one-sided 0.0177
Z-score
2.103
SE 0.0143
Observed power
55.77%
post-hoc
95% CI (difference)
0.21 … 5.79 pp
+3.00 pp observed
Sampling distributions of the conversion rate
Distribution of the difference (H₀: no effect)
Shaded = rejection region at 95% confidence. The observed difference falling inside it means significance.
How the statistics work
This calculator runs a two-proportion z-test— the standard method for comparing conversion rates between two independent groups. It estimates each variant's conversion rate, computes the standard error of the difference, and asks: if both variants truly performed the same, how unusual would the observed gap be?
z = (p̂B − p̂A) / SE, SE = √( p̂A(1−p̂A)/nA + p̂B(1−p̂B)/nB )
The answer is the p-value. When it drops below your significance threshold (5% at a 95% confidence level), the difference is unlikely to be noise and the result is called statistically significant. The confidence interval shows the plausible range of the true difference, and observed power tells you whether the test was even capable of detecting an effect of this size.
Statistical significance alone doesn't make a result safe to ship: broken traffic splits (SRM), declining guardrail metrics, or a segment moving against the global effect can all turn a "winner" into a costly mistake. That's the decision layer Converise adds on top of calculators like this one. Planning a test instead? Start with the sample size calculator.
Frequently asked questions
How does this A/B test significance calculator work?
It runs a two-proportion z-test on your numbers: it compares the conversion rate of the variant against the control, computes the standard error of the difference, and derives the z-score, p-value, and confidence interval. If the p-value is below your significance threshold (1 − confidence level), the result is statistically significant.
What is a p-value?
The p-value is the probability of seeing a difference at least as large as the one you observed if there were truly no difference between the variants (the null hypothesis). A p-value of 0.03 at 95% confidence means: if A and B actually performed identically, a gap this size would appear in fewer than 3 out of 100 identical experiments.
Should I use a one-sided or two-sided test?
Two-sided is the safer default: it tests whether B is different from A in either direction. Use one-sided only when you decided before the test that you exclusively care about one direction — it halves the p-value, so choosing it after seeing the data inflates false positives.
What is a sample ratio mismatch (SRM)?
SRM means the traffic split between control and variant deviates from the expected 50/50 more than chance allows. It usually signals broken randomization, bot filtering, or tracking loss — and it invalidates the test result regardless of the p-value. This calculator flags SRM automatically.
What does observed power mean?
Power is the probability that a test detects an effect of the observed size, given your sample. Low observed power (below ~80%) means a real effect of that size could easily have been missed — treat non-significant results from underpowered tests as inconclusive, not as proof of no effect.
Is this calculator free? Where do my numbers go?
It is completely free, with no signup. All calculations run in your browser — nothing you type is stored or sent to any server.
Significance is step one. Decisions are the product.
Converise checks every experiment for SRM, guardrail declines, and segment conflicts before you ship — then remembers the decision so your team never repeats a failed test.
Try Converise free