Public evidence audit · reviewed July 23, 2026
Which A/B test calculators can analysts actually trust?
We compared planning, analysis, statistical controls, SRM, effect intervals, practical decisions, sharing, export, and the handoff into an experiment repository. The dataset is public so you can inspect—and challenge—the findings.
- public calculators reviewed
- 11
- workflow capabilities compared
- 13
- opaque quality scores assigned
- 0
Strongest calculator-only UX
Convert is the benchmark to beat on statistical breadth.
It was the broadest observed public calculator: planning and analysis, multiple metric types, SRM, intervals, and frequentist, sequential, and Bayesian methods. That makes the old claim that every competitor is fragmented inaccurate.
GrowthLayer differentiation
The advantage is the analyst handoff, not “more math.”
GrowthLayer is the reviewed workflow that joins fixed-horizon planning and analysis with shareable state, analyst copy, CSV, and a direct path into a private experiment repository. Convert remains broader on method choice.
Capability matrix
What each public calculator actually supports
Scroll horizontally on smaller screens. “Not verified” is deliberately different from “No.”
YesPartialNoNot verified
| Calculator | Workflow | Pre-test planning | Post-test analysis | Statistical method choice | One- or two-sided choice | Relative and absolute MDE | Unequal arm allocation | Experiment exposure control | Sample ratio mismatch check | Effect-size interval | Practical decision support | Shareable calculator state | Analyst-ready copy or export | Experiment repository handoff |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Convertobserved | planning and analysis | Yes | Yes | Yes | Yes | Yes | Not verified | Yes | Yes | Yes | Yes | No | No | No |
| GrowthLayerobserved | planning and analysis | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
| Speerolimited access | limited access | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified | Not verified |
| ABTestGuideobserved | planning and analysis | Yes | Yes | No | Yes | Not verified | No | No | Yes | No | Partial | Yes | No | No |
| SurveyMonkeysource review | analysis only | No | Yes | No | Yes | Not verified | No | No | No | No | Not verified | No | No | No |
| Kameleoonsource review | multi tool page | Yes | Partial | Not verified | Not verified | Not verified | Not verified | Not verified | No | Not verified | Partial | No | No | No |
| Evan Millerobserved | planning only | Yes | No | No | Yes | Yes | No | No | No | No | No | Yes | No | No |
| AB Tastysource review | planning only | Yes | No | No | No | Partial | No | No | No | No | Not verified | No | No | No |
| Optimizelysource review | planning only | Yes | No | No | No | Partial | No | No | No | No | No | Not verified | No | No |
| VWOsource review | planning only | Yes | Partial | Partial | Not verified | Partial | Not verified | Not verified | No | Not verified | Yes | No | No | No |
| AtticusLiobserved | planning and analysis | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Partial |
Reproducible method
How the audit was run
Manual review of public calculator controls, outputs, help copy, share behavior, and first-party methodology pages using a common fixed-horizon planning fixture where supported.
- baseline Conversion Rate
- 5%
- minimum Detectable Effect
- 10% relative (5.0% to 5.5%)
- alpha
- 0.05
- power
- 0.8
- tail
- two-sided
- allocation
- 50/50
Evidence limits
What this dataset does not claim
- This audit covers public calculator pages and public documentation, not every capability inside a paid experimentation platform.
- No composite quality score is assigned. Statistical approaches solve different decision problems, so feature count is not a validity ranking.
- Not verified means the capability could not be confirmed from the reviewed public surface. It does not mean the provider lacks it.
- A page that could not be loaded is treated as an evidence limitation, never as a product-quality failure.
- Capabilities and copy can change. Use the review date and first-party links when citing the dataset.
Provider notes
One strength and one limitation for every calculator
Convert
planning and analysis- Primary strength
- The broadest observed calculator-only workflow, with frequentist, sequential, and Bayesian options.
- Material limitation
- No public share, copy, download, or experiment-repository handoff was observed after calculation.
- Evidence note
- Public calculator and its on-page methodology were reviewed in a desktop browser.
GrowthLayer
planning and analysis- Primary strength
- One fixed-horizon workflow connects planning, analysis, SRM, sharing, export, and experiment storage.
- Material limitation
- The unified planner does not let the analyst switch among frequentist, sequential, and Bayesian engines.
- Evidence note
- Public calculator behavior, output, share state, CSV, and repository CTA were exercised.
Speero
limited access- Primary strength
- The first-party wrapper describes an experimentation calculator and companion Chrome extension.
- Material limitation
- The embedded calculator could not be loaded from the audit environment, so its capabilities were not scored.
- Evidence note
- Access failure is an evidence limitation, not a negative product-quality finding.
ABTestGuide
planning and analysis- Primary strength
- Combines a compact planner, post-test calculation, SRM, and shareable state.
- Material limitation
- Its confidence explanation can be read as a probability that the observed consequence is not random.
- Evidence note
- Public calculator controls, result language, and share behavior were reviewed.
SurveyMonkey
analysis only- Primary strength
- A simple post-test significance check with a visible tail choice.
- Material limitation
- The surrounding copy describes confidence and p-values in probability-of-chance language that is not statistically precise.
- Evidence note
- The first-party calculator page, instructions, and definitions were reviewed.
Kameleoon
multi tool page- Primary strength
- Traffic, duration, MDE, and power questions are collected on one page.
- Material limitation
- The tools remain separate, and the FAQ equates a 95% interval with only a 5% chance that results are random.
- Evidence note
- First-party calculator modules and explanatory FAQ copy were reviewed.
Evan Miller
planning only- Primary strength
- Transparent, reproducible planning math with relative and absolute MDE.
- Material limitation
- It is a focused pre-test reference, not an experiment decision workflow.
- Evidence note
- The public sample-size calculator, controls, notes, and share link were reviewed.
AB Tasty
planning only- Primary strength
- Turns baseline, MDE, traffic, and variant count into sample and duration guidance.
- Material limitation
- The public page is pre-test only and fixes important analytical choices.
- Evidence note
- First-party calculator inputs, outputs, and supporting guidance were reviewed.
Optimizely
planning only- Primary strength
- A low-friction planning check aligned to Optimizely Stats Engine users.
- Material limitation
- It is not a public end-to-end analysis workflow, and results are engine-specific.
- Evidence note
- The first-party public planner and Stats Engine framing were reviewed.
VWO
planning only- Primary strength
- Supports conversion and revenue planning, plus SmartStats decision concepts such as practical equivalence.
- Material limitation
- The public planner and separate significance tool do not form one shareable analyst handoff.
- Evidence note
- First-party planner, SmartStats descriptions, and related tool links were reviewed.
AtticusLi
planning and analysis- Primary strength
- A consulting trust surface with planning, analysis, SRM, share links, analyst copy, and CSV.
- Material limitation
- It refers users to GrowthLayer rather than storing the calculation with an experiment itself.
- Evidence note
- Public calculator controls, output, exports, and GrowthLayer referral were reviewed.
From calculation to institutional memory
Plan the test, analyze it, then preserve the decision.
GrowthLayer’s unified calculator creates a shareable analyst handoff. The repository keeps the hypothesis, evidence, result, and decision together for later review.
First-party sources
Provider pages were reviewed on 2026-07-23.