Free Calculator
Bayesian A/B Test Calculator
Estimate directional probability, credible intervals, practical effect, harm, and expected loss under a declared Bayesian model.
Required. GrowthLayer does not assume which direction is desirable.
Required. Also used as the meaningful-harm boundary for this sensitivity view.
GrowthLayer does not invent a universal minimum.
Beta(1,1) — equal density across rates. Choose this before reviewing the result.
This is a sensitivity analysis, not a stopping rule.
Before launch, declare the prior, practical-effect or expected-loss threshold, and when the test may stop. Repeatedly checking this probability and stopping when it looks favorable changes the decision process.
Check your inputs
Choose whether higher or lower values are better for your metric.
You just ran the numbers. Where will the result live?
A calculator answers one question and forgets it. GrowthLayer keeps each test your team saves — the hypothesis, the numbers, and the decision — so future planning can start from a reviewable record.
- Structured history of saved experiments
- Import results from CSV in one step
- Program summaries across saved tests
- Free to start — no card required
Methodology
This calculator uses the Beta-Binomial conjugate model, the standard Bayesian approach for analyzing conversion rate experiments.
For each variant, conversions follow a Binomial distribution, and the conversion rate follows a Beta prior. After observing data, the posterior distribution for each variant's conversion rate is:
Posterior_A = Beta(alpha + conversions_A, beta + visitors_A - conversions_A)
Posterior_B = Beta(alpha + conversions_B, beta + visitors_B - conversions_B)
Where alpha and beta are the prior parameters. With a weak prior (alpha=1, beta=1), the posterior is essentially determined by the data.
This page provides two weak sensitivity priors: uniform Beta(1,1) and Jeffreys Beta(0.5,0.5). It does not infer an informative prior from the observed test. You can declare the desired metric direction, the smallest useful change, and a minimum visitor count. These fields make the directional readout clearer, but they do not recreate a stopping rule after the test has ended.
Posterior probabilities are not a license for unrestricted optional stopping. A Bayesian-primary decision also needs a prior, acceptable loss, minimum data, maximum duration, monitoring cadence, stopping rule, multiplicity treatment, and calibration saved before results are reviewed. This calculator therefore reports sensitivity evidence only.
The probability that B beats A is estimated using Monte Carlo simulation with 10,000 draws from each posterior distribution. For each simulation:
1. Draw a sample conversion rate from Posterior_A
2. Draw a sample conversion rate from Posterior_B
3. Record whether B > A and the difference
From these simulations we compute:
- P(B > A): fraction of draws where B's sample exceeded A's
- Observed absolute and relative lift
- Posterior median absolute and relative effect
- 95% credible intervals calculated directly from the corresponding posterior effect samples
- Probability of any improvement, a practically useful improvement, and meaningful harm
- Expected conversion-rate loss from choosing control or variant
Beta distribution samples are generated using Marsaglia & Tsang's method for Gamma variates, with a seeded PRNG for reproducibility.
Frequently Asked Questions
What is the difference between Bayesian and frequentist A/B testing?
Frequentist testing asks: 'If there is no real difference, how compatible are these data with that assumption?' Bayesian testing asks: 'Given the observed data and the declared prior, how likely is the metric to move in the desired direction?' Either approach needs a decision rule chosen before results are used to stop the test. Bayesian updating does not, by itself, make unrestricted peeking or optional stopping harmless.
When should I use a Bayesian A/B test calculator?
Use this calculator when you want a direct posterior probability and expected-loss sensitivity analysis. Before launch, document the prior, practical-effect threshold, acceptable loss, and stopping rule. If those were not predeclared, treat the output as exploratory rather than as a reconstructed decision analysis.
What is a credible interval?
A 95% credible interval contains 95% of the posterior effect distribution under the selected model and prior. It describes the range of effects still plausible after combining that prior with the observed data.
What does the prior strength setting do?
The prior represents assumptions about the conversion rate before seeing the current test. This calculator offers only two weak reference priors: uniform Beta(1,1) and Jeffreys Beta(0.5,0.5). It does not offer arbitrary stronger priors centered at 50%. A genuinely informative prior should come from relevant historical data and be justified before the test begins.
What does 'risk of choosing' mean?
Expected loss estimates the average conversion-rate opportunity cost of each choice under the selected model and prior. A value of 0.5 percentage points for choosing control means the posterior-average loss from cases where the variant is better is 0.5 points. It is decision support, not proof that either choice is safe.
Related Calculators
A/B Test Significance CalculatorSample Size CalculatorMinimum Detectable Effect CalculatorChi-Squared Test Calculator
Choosing a method or double-checking another tool? Compare 11 public A/B test calculators and download the evidence matrix.
Updated for 2026. Built by GrowthLayer — built for evidence-aware experimentation teams.