I Audited 11 A/B Test Calculators: Which Ones Can Analysts Trust?

Most “A/B test calculators” solve one small equation and leave the analyst to assemble the decision somewhere else. One page estimates sample size. Another checks significance. A third checks sample ratio mismatch. The assumptions often change between pages, and the handoff is a screenshot pasted into a testing platform.

I audited 11 widely used calculators from the perspective of a statistician, a CRO practitioner, and the analyst who has to defend the result in a meeting. The winner is not the page with the most inputs. It is the workflow that makes assumptions visible, keeps planning and analysis compatible, and helps the team make the next decision without overstating the evidence.

How I evaluated them

I used the same core planning fixture wherever the tool supported it:

• Baseline conversion rate: 5.0%
• Minimum detectable effect: 10% relative, or 5.0% → 5.5%
• Alpha: 0.05
• Power: 80%
• Direction: two-sided
• Allocation: 50/50

For the standard pooled two-proportion planning formula, the reproducible answer is 31,234 observations per arm. A different answer is not automatically wrong. Continuity corrections, unequal allocation, alternative tests, sequential monitoring, and Bayesian decision rules can all change the requirement. The problem is unexplained disagreement, not disagreement itself.

I compared each public experience against ten practical questions. I did not create a composite score because feature count is not statistical validity:

1. Can it plan the experiment before launch?
2. Can it analyze the result after launch?
3. Are the statistical method and stopping rule clear?
4. Can the analyst choose one- versus two-sided inference?
5. Can MDE be entered and reported as both relative and absolute change?
6. Does it support allocation and exposure instead of assuming every visitor sees a 50/50 test?
7. Does it check sample ratio mismatch before celebrating significance?
8. Does it report an effect interval and practical impact, not only a threshold label?
9. Can the state or analyst-ready result be shared and exported?
10. Can the calculation be preserved with the experiment, or does the workflow end at a download?

The full A/B test calculator audit dataset publishes all 13 capability fields as CSV and JSON. It uses four evidence statuses: yes, partial, no, and not verified. “Not verified” matters because a calculator that could not be loaded should not receive a negative product-quality score.

The short answer

Convert: the strongest calculator-only benchmark

Convert's unified calculator materially changes the competitive picture. The public workflow supports planning and analysis for conversion rate, revenue per visitor, and products per visitor. It exposes frequentist, group-sequential, and Bayesian approaches; multiple variants; SRM; confidence or credible intervals; power; and expected-loss concepts.

That makes Convert the strongest observed calculator-only benchmark in this audit. The limitation is the operating handoff: after calculating, I did not observe a public share-state button, analyst-summary copy, CSV download, or path that preserved the calculation with an experiment record. GrowthLayer should learn from Convert's statistical breadth rather than pretend it does not exist.

Speero: popular, but not fairly scoreable in this audit

Speero is widely referenced by CRO teams, and its first-party wrapper describes a calculator plus a browser extension. However, the embedded calculator returned a connection failure from the audit environment. That means I could verify the wrapper's claims, but not exercise the calculator controls or outputs.

The correct evidence label is not verified, not “bad” or “missing.” Access failure measures the audit environment, not product quality. This distinction is why the downloadable dataset avoids a simplistic winner score.

Evan Miller: the best reproducible reference

Evan Miller's sample-size calculator remains the reference I use to sanity-check conventional fixed-horizon planning. The method is documented, the interface is fast, and an analyst can reproduce the answer independently.

Its limitation is product scope, not statistical credibility. It helps answer “how much sample do I need?” It does not check the actual allocation, interpret the final effect interval, compare the result with the pre-registered MDE, or produce an analyst handoff. It is an excellent ruler, not an experiment operating system.

VWO and Optimizely: do not compare unlike engines

VWO's public sample-size calculator is useful for conventional planning. VWO's product analysis has historically used its own SmartStats approach. Those outputs answer different probability questions, so the public planner should not be treated as a complete description of the platform engine.

Optimizely's sample-size guidance belongs to a sequential testing ecosystem. Optimizely Stats Engine is designed for ongoing monitoring under its own rules. A fixed-horizon sample-size result and a sequential stopping boundary are not interchangeable. If you use Optimizely, plan and analyze within the same declared engine rather than mixing its output with a conventional significance page.

The weakness here is not that proprietary or sequential methods are bad. It is that marketing language often makes unlike numbers look directly comparable.

AB Tasty and Kameleoon: useful planning, incomplete lifecycle

AB Tasty turns baseline, MDE, traffic, and variant count into sample and duration guidance. Kameleoon collects traffic-and-duration, MDE, and power questions on one page. Both can help a marketer assess feasibility without starting in a statistics package.

Their public workflows are less complete once the test ends. A production decision also needs allocation validation, observed absolute and relative effects, an interval, the planned sample, and a statement about whether the result is practically large enough to matter. Kameleoon's FAQ also says a 95% confidence interval means a 5% chance results are random; that is not what a frequentist confidence interval means.

ABTestGuide: more complete than its simple interface suggests

ABTestGuide combines planning and post-test analysis, lets the user choose the tail, checks SRM, and creates a shareable URL. That is more operationally useful than many better-known single-purpose calculators.

The gaps are the effect-size interval, analyst-ready export, and durable experiment record. Its result language also describes 95% confidence as confidence that a consequence is not random, which invites the same 1 − p interpretation the industry should retire.

SurveyMonkey: simple post-test check, incomplete decision

SurveyMonkey's significance calculator is approachable for someone who already has visitor and conversion counts. It reduces the intimidation of a statistical test.

But it begins after the most important design choices should have been made. There is no pre-test power plan, allocation diagnosis, planned stopping rule, or durable link between the hypothesis and the result. Beginner-friendly language is useful only if it remains precise: a p-value is not the probability the result happened by chance, and 1 − p is not the probability that the variant wins.

What was wrong with the old AtticusLi and GrowthLayer tools

The former AtticusLi sample-size calculator was fast but too narrow. It supported pre-test binary conversion planning only, silently capped impossible target rates, duplicated its math in the browser, offered no tail or allocation choice, and made an inaccurate claim that every major platform used the same math.

The earlier GrowthLayer tools had the opposite problem: enough individual calculators existed, but sample size, significance, duration, MDE, revenue, and SRM lived on separate pages. The significance page also presented 1 − p-value as “confidence,” which is not a valid interpretation.

Both lessons shaped the new unified GrowthLayer A/B test calculator:

• Plan and Analyze live in one workflow.
• Conversion and continuous average metrics use appropriate methods.
• Two-sided is the default, with an explicit one-sided option.
• Relative and absolute MDE, unequal allocation, and exposure are visible.
• Impossible conversion targets produce an error instead of a hidden cap.
• Post-test interpretation starts with SRM, then effect and interval, then p-value.
• Planned sample and practical threshold affect the decision language.
• A share link, analyst summary, and CSV take one click.
• The next action is saving the experiment—not opening another calculator.

The focused sample-size calculator, significance calculator, and SRM calculator remain available for analysts who need to verify one narrow result.

The AtticusLi version now uses the same fixed-horizon math and supports Plan and Analyze, conversion and continuous metrics, SRM, intervals, share links, analyst copy, and CSV. It remains intentionally lightweight as a consulting trust asset and referral surface. GrowthLayer is the canonical operating tool because the calculation can lead naturally into a searchable experiment record.

The calculator I would standardize on

If I only needed an independent fixed-horizon planning check, I would use Evan Miller. If I wanted the broadest public calculator-only workflow, I would evaluate Convert. If my organization already runs Optimizely, VWO, or another proprietary engine, I would follow that engine end to end.

For a team standard, I want one transparent workflow that remembers the plan, diagnoses the data before interpreting it, communicates uncertainty, and then preserves the result. GrowthLayer currently makes a deliberate trade: less method choice than Convert, but a tighter fixed-horizon analyst handoff into the experiment repository. The audit data makes that tradeoff visible instead of hiding it behind a “best calculator” badge.

The statistical method matters. The operating behavior matters more. A calculator that returns 31,234 instead of 31,600 will not save a program that peeks, ignores SRM, changes metrics after launch, or loses every result in a spreadsheet.

FAQ

Which A/B test calculator is most accurate?

There is no universal winner without naming the analysis method. A sample-size formula should match the test and stopping rule used for analysis. For a conventional fixed-horizon two-proportion plan, Evan Miller is a strong reproducible reference. For an operating workflow, choose a tool that keeps planning, analysis, and documentation compatible.

Why do two calculators return different sample sizes?

They may use different tail choices, continuity corrections, allocation assumptions, multiple-comparison corrections, or entirely different fixed-horizon, sequential, or Bayesian methods. Compare the assumptions before comparing the final numbers.

Is a p-value of 0.03 equal to 97% confidence?

No. The p-value is calculated under the null hypothesis. Subtracting it from one does not produce the probability that the variant wins or that the result is true.

Should a post-test calculator check SRM?

Yes. Unexpected allocation can indicate implementation or data problems. A result that fails a strong SRM check should be investigated before the outcome p-value is used for a ship decision.

Sources and calculators reviewed

• Speero on A/B test calculator features
• Downloadable calculator audit dataset
• ABTestGuide calculator
• SurveyMonkey A/B significance calculator
• Kameleoon A/B testing calculator
• Evan Miller sample-size calculator
• AB Tasty sample-size calculator
• Convert unified calculator
• Optimizely sample-size calculator
• VWO sample-size calculator
• NIST comparison of two proportions
• Casagrande, Pike, and Smith (1978)