Free Calculator
Statistical Significance Calculator for A/B Tests
Analyze conversion rates or continuous averages with an SRM-first fixed-horizon workflow. Get the p-value, effect-size interval, practical decision, share link, analyst summary, and CSV.
Enter your observed results
Result fields start blank so example numbers cannot be mistaken for your experiment.
Statistical settings and original plan
Use the assumptions declared before launch. A plan created in this calculator is carried here automatically.
Ready for your results
Enter both arms above, or load the clearly labeled example, to generate a decision readout.
Share links store inputs in the URL fragment, which is not sent to the server. Anyone with the link can read those inputs.
The p-value is one line. Preserve the decision behind it.
A calculator answers one question and forgets it. GrowthLayer keeps each test your team saves — the hypothesis, the numbers, and the decision — so future planning can start from a reviewable record.
- Structured history of saved experiments
- Import results from CSV in one step
- Program summaries across saved tests
- Free to start — no card required
Methodology
This is a fixed-horizon frequentist calculator for two independent experiment arms.
For binary conversion metrics, the outcome p-value comes from a pooled two-proportion z-test. The effect is reported as both an absolute percentage-point difference and a relative lift. The uncertainty interval uses Newcombe's hybrid score method 10: separate Wilson score intervals are combined into an interval for the difference in proportions.
For continuous average metrics, the calculator uses Welch's unequal-variance t approximation for the standard error and degrees of freedom. The effect and interval remain in the original metric unit.
Before interpreting either outcome, the calculator runs a chi-square goodness-of-fit SRM check against the expected traffic allocation. GrowthLayer uses p < 0.001 as the default strong warning. SRM is a diagnostic alarm, not proof of one specific implementation bug.
The selected alpha controls the p-value threshold and interval level. Two-sided is the recommended default; the one-sided option is increase-only and must be chosen before inspecting results. Planned samples and the smallest useful effect are optional inputs that change the decision language.
Do not use this fixed-horizon result for repeated optional stopping. Observations must be independent at the randomization unit, conversions cannot exceed visitors, standard deviations must be positive, and the metric definition must be identical across arms.
Sources and limitations
- NIST: comparing two proportions — Reference for large-sample comparison of two independent proportions.
- Newcombe (1998): interval estimates for differences between proportions — Source for the score-based difference interval used for binary metrics.
- NIST: two-sample t-test — Reference for comparing independent means; this calculator uses the unequal-variance form.
- Microsoft Research: diagnosing sample ratio mismatch — Explains SRM root causes and why allocation integrity comes before outcome interpretation.
Frequently Asked Questions
What does statistical significance mean in A/B testing?
A fixed-horizon result is statistically significant when its p-value crosses the alpha declared before analysis and the test assumptions hold. That describes a long-run error-control procedure. It does not mean there is a 95% probability the variant is better.
What is an A/B test p-value?
The p-value is the probability, under the null hypothesis and model assumptions, of a result at least as extreme as the one observed. It is not the probability the result happened by chance, and 1 minus the p-value is not confidence that the variant wins.
Why does the calculator check sample ratio mismatch first?
Unexpected allocation can signal problems in assignment, exposure logging, redirects, filtering, bots, or analysis. Investigate a strong SRM warning before interpreting the outcome effect because a small outcome p-value cannot repair biased input data.
Should I use a one-sided or two-sided A/B test?
Two-sided is the safer default because a variant can help or harm. A one-sided increase-only test can be justified only when declared before seeing results, effects in the opposite direction would not be called significant, and the decision process genuinely matches that constraint.
What confidence interval does this calculator use?
For conversion rates, it uses a Newcombe hybrid score interval for the difference between two independent proportions. For continuous averages, it uses a Welch interval, which does not assume equal variance. The interval level follows the selected alpha.
Can I stop when the calculator first shows significance?
Not in an ordinary fixed-horizon workflow. Repeated checking and stopping at the first favorable threshold changes the false-positive behavior. Use the declared final sample or a valid sequential procedure designed for repeated looks.
Does a significant result mean I should ship?
No. Review SRM, sample-plan completion, effect direction, uncertainty interval, guardrails, and whether the effect clears the smallest useful business threshold. A statistically detectable result can still be too small or risky to ship.
Related Calculators
Unified A/B Test CalculatorSample Ratio Mismatch CalculatorA/B Test Sample Size CalculatorA/B Test Duration Calculator
Choosing a method or double-checking another tool? Compare 11 public A/B test calculators and download the evidence matrix.
Updated for 2026. Built by GrowthLayer — built for evidence-aware experimentation teams.