Skip to main content
Free Calculator

Bayesian A/B Test Calculator

Calculate the probability that your variant beats control using Bayesian statistics. Get credible intervals, expected lift, and risk analysis in real time.

Beta(1,1) — lets the data speak for itself

Results

Probability Variant Wins

92.0%

Control winsVariant wins
8.0%92.0%

Expected Lift

+31.58%

relative improvement

95% Credible Interval

[-12.03%, +71.22%]

lift range

Risk of Choosing Control

1.524 pp

expected loss in conversion rate

Risk of Choosing Variant

0.038 pp

expected loss in conversion rate

Methodology

This calculator uses the Beta-Binomial conjugate model, the standard Bayesian approach for analyzing conversion rate experiments. For each variant, conversions follow a Binomial distribution, and the conversion rate follows a Beta prior. After observing data, the posterior distribution for each variant's conversion rate is: Posterior_A = Beta(alpha + conversions_A, beta + visitors_A - conversions_A) Posterior_B = Beta(alpha + conversions_B, beta + visitors_B - conversions_B) Where alpha and beta are the prior parameters. With a weak prior (alpha=1, beta=1), the posterior is essentially determined by the data. The probability that B beats A is estimated using Monte Carlo simulation with 10,000 draws from each posterior distribution. For each simulation: 1. Draw a sample conversion rate from Posterior_A 2. Draw a sample conversion rate from Posterior_B 3. Record whether B > A and the difference From these simulations we compute: - P(B > A): fraction of draws where B's sample exceeded A's - Expected lift: average relative improvement of B over A - 95% credible interval: 2.5th and 97.5th percentiles of the lift distribution - Risk: expected conversion rate loss from choosing the wrong variant Beta distribution samples are generated using Marsaglia & Tsang's method for Gamma variates, with a seeded PRNG for reproducibility.

Frequently Asked Questions

What is the difference between Bayesian and frequentist A/B testing?▾
Frequentist testing asks: 'If there is no real difference, how likely is the observed data?' It gives you a p-value and requires a fixed sample size decided in advance. Bayesian testing asks: 'Given the observed data, how likely is variant B better than control?' It gives you a direct probability of one variant beating another, and you can check results at any time without inflating error rates.
When should I use a Bayesian A/B test calculator?▾
Use Bayesian testing when you want intuitive results (direct probability of winning), when you need to check results before the test reaches a fixed sample size, when you want to quantify the risk of choosing the wrong variant, or when you want to incorporate prior knowledge about expected conversion rates.
What is a credible interval?▾
A credible interval is the Bayesian equivalent of a confidence interval. A 95% credible interval means there is a 95% probability that the true difference in conversion rates falls within this range, given the observed data and prior. Unlike frequentist confidence intervals, credible intervals have a direct probabilistic interpretation.
What does the prior strength setting do?▾
The prior represents your belief about the conversion rate before seeing any data. A weak prior (Beta(1,1)) is uniform and lets the data speak for itself — best when you have no prior knowledge. A medium prior (Beta(5,5)) gently centers around 50% and smooths out noise in small samples. A strong prior (Beta(50,50)) heavily centers around 50% and requires substantial data to move — use this only if you have strong historical evidence.
What does 'risk of choosing' mean?▾
Risk (or expected loss) quantifies what you stand to lose by choosing the wrong variant. If the risk of choosing A is 0.5 percentage points, it means that on average you would lose 0.5 percentage points of conversion rate by picking A when B is actually better. Low risk on both sides means the variants are very similar and either choice is safe.

Related Calculators

A/B Test Significance CalculatorSample Size CalculatorMinimum Detectable Effect CalculatorChi-Squared Test Calculator

Updated for 2026. Built by GrowthLayer.