9 A/B Testing Calculator Mistakes That Create False Winners

An A/B testing calculator can return a perfectly accurate number and still support the wrong decision. The arithmetic may be correct while the analyst uses the wrong baseline, changes the stopping rule, ignores broken allocation, or mistakes statistical significance for business value. These failures are dangerous because the result looks precise enough to defend in a meeting.

The calculator is not the experiment. It is one checkpoint in a system that begins with the decision you need to make and ends with evidence another analyst can reproduce. Here are the nine calculator mistakes I would fix before debating whether one sample-size formula is slightly better than another.

1. Choosing MDE from hope instead of business value

Minimum detectable effect is the smallest effect your plan is designed to detect with the declared power. It is not a forecast. Entering a 20% relative lift because the redesign feels substantial does not make that lift likely.

Start with the smallest effect that would change the decision. If a 3% relative improvement would not cover engineering cost, operational risk, or opportunity cost, powering the test for 3% buys precision the business will not use. If a 3% lift would be valuable, choosing a 15% MDE merely to make the test fit a short calendar hides a real feasibility problem.

Use the sample-size calculator to make the tradeoff visible, then save both the relative and absolute effect. A 10% relative lift on a 5% baseline is only 0.5 percentage points.

2. Treating sample size per arm as the total

Many calculators report the required sample for each variation. An A/B test therefore needs roughly twice that number in total. An A/B/C/D test needs traffic for four cells, and it may also need a multiple-comparison adjustment.

This error usually appears in the duration estimate. The analyst sees “31,000 required” and divides it by total daily traffic. If the number is per arm and traffic is split evenly, the resulting calendar estimate can be half the required runtime.

Write the unit beside every result:

• observations per arm;
• total observations;
• eligible daily traffic;
• experiment exposure percentage; and
• allocation for every arm.

The unified A/B test planner keeps allocation and exposure next to the sample result so the unit cannot quietly change between a planning spreadsheet and the launch brief.

3. Using a stale or blended baseline

Sample requirements depend on the control conversion rate and variance. A baseline from last year's holiday campaign, all devices combined, or the entire site may not describe the eligible population in the planned experiment.

Pull the baseline from the same metric definition, eligibility rule, market, device scope, and randomization unit the test will use. Include enough history to cover the normal weekly cycle, but investigate major shifts instead of averaging them away.

The goal is not to find the one “true” conversion rate. It is to describe the population you plan to randomize. If uncertainty is material, calculate a plausible baseline range and plan from the conservative scenario rather than choosing whichever input produces the shortest test.

4. Mixing relative and absolute lift

“A five percent lift” is ambiguous. It can mean a 5% relative change or five percentage points.

At a 10% baseline:

• 5% relative means moving from 10% to 10.5%;
• five percentage points means moving from 10% to 15%.

Those are radically different experiments with radically different sample requirements. The mistake often survives because a calculator labels the field “expected improvement” without showing the target rate.

A trustworthy planning workflow should display baseline, unit, and target together. If the target would exceed 100% or fall below 0%, it should show an error instead of silently clipping the value. Copy the absolute and relative forms into the experiment plan so a later analyst does not have to infer what the original team meant.

5. Planning with one method and analyzing with another

There is no universal A/B testing sample-size formula. The planning calculation should correspond to the hypothesis test, stopping rule, and multiple-comparison policy used in analysis.

A conventional fixed-horizon two-proportion plan assumes a declared final sample and a compatible fixed-horizon analysis. A group-sequential procedure spends error across planned looks. A Bayesian workflow uses priors and decision thresholds. These approaches can all be defensible, but their outputs are not interchangeable.

Do not take a sample target from a frequentist calculator, peek until a platform's sequential result looks favorable, and validate the winner in a third significance tool. Pick a coherent method end to end. The public calculator audit shows which reviewed tools expose method choice and which are intentionally fixed-horizon.

6. Peeking and stopping at the first favorable threshold

If a fixed-horizon plan says to analyze after 60,000 eligible observations, checking every morning and stopping when p drops below 0.05 changes the error behavior. The nominal threshold no longer describes the actual repeated decision process.

The solution is not “never look at experiments.” Teams should monitor implementation health, guardrails, safety, and allocation. The restriction applies to repeatedly making the success decision from an ordinary fixed-horizon p-value.

Choose one of three honest policies:

1. Declare the final sample and make the outcome decision there.
2. Declare a small number of interim looks and use a valid group-sequential design.
3. Use a continuously valid or Bayesian procedure with its own documented decision rule.

The A/B test duration calculator can translate a fixed-horizon sample into a calendar estimate. It cannot make optional stopping valid.

7. Reading the outcome before checking SRM

Sample ratio mismatch means observed assignment counts differ more from the planned allocation than the declared diagnostic threshold allows. It is a data-quality alarm, not proof of one particular bug.

An SRM can come from randomization, exposure logging, redirects, filtering, missing events, bots, or analysis choices. If those mechanisms affect who appears in each arm, a tiny outcome p-value does not rescue the comparison.

Check allocation before interpreting the treatment effect. Use a strict threshold declared in advance, inspect the split over time, and investigate the original randomization unit. A final aggregate 50/50 split can hide a period-specific failure.

The sample ratio mismatch calculator supports unequal plans and multiple arms, but it is a fixed-horizon snapshot. Repeated SRM monitoring also needs a sequential policy.

8. Reporting p-value without effect and interval

A p-value is calculated under a null hypothesis. It is not the probability that the result happened by chance, and 1 − p is not the probability that the variant wins.

Outcome reporting should begin with the effect:

• control and variant rates or means;
• absolute change;
• relative change;
• uncertainty interval; and
• the business threshold that would change the decision.

Then report the p-value as one piece of evidence. A tiny but commercially irrelevant effect can become significant with enough traffic. A promising effect with a wide interval may need more evidence. A non-significant result does not prove equivalence unless the design and interval support an equivalence claim.

Use the statistical significance calculator for a focused check, but carry the interval and practical decision into the experiment record.

9. Losing the assumptions after copying the answer

The most common calculator output is a screenshot. It proves that someone entered something into a tool at some time. It rarely preserves the inputs, formula, tail, allocation, stopping rule, or the reason those choices were made.

A defensible analyst handoff should include:

• metric and randomization unit;
• baseline and MDE in both units;
• alpha, power, tail, and method;
• allocation, exposure, and variant count;
• target sample and duration assumptions;
• final counts and observed effects;
• SRM result and diagnostic threshold;
• effect interval and p-value;
• practical threshold and decision; and
• calculator version or share URL.

One-click copy and CSV export reduce transcription errors. The next step should be saving the plan and result with the experiment, not opening an unrelated calculator and rebuilding the context.

A better calculator workflow

The sequence matters more than the number of fields:

1. Define the decision and smallest worthwhile effect.
2. Plan sample and duration using the declared analysis method.
3. Save the assumptions before launch.
4. Monitor implementation and guardrails without optional outcome stopping.
5. Check allocation integrity.
6. Report effect and interval before threshold labels.
7. Compare the observed evidence with the planned practical threshold.
8. Save the decision, caveats, and analyst handoff with the experiment.

That is why GrowthLayer's calculator strategy is not “put every equation on one page.” Convert currently offers broader public method choice. Evan Miller remains an excellent independent planning reference. GrowthLayer's job is to create a calm fixed-horizon workflow and carry its evidence into searchable experiment memory.

FAQ

What is the biggest A/B testing calculator mistake?

Using a statistically valid calculation inside an invalid decision process. Common examples are peeking at a fixed-horizon test, changing the primary metric after launch, or interpreting outcomes before investigating an SRM.

Is 95% confidence the probability that my variant wins?

No. A conventional confidence level describes the long-run coverage behavior of the interval procedure under repeated sampling and its assumptions. It is not a posterior probability that this specific variant is better.

Should every A/B test run for two weeks?

No universal calendar minimum replaces sample and power. Running complete weekly cycles can reduce day-of-week distortion, but the test also needs adequate eligible sample and a stable business environment. Use the sample requirement and traffic to estimate duration, then account for cycles and operational risk.

Can I use one calculator for planning and another for significance?

Only after confirming that the methods, tail, stopping rule, and corrections are compatible. An unexplained match in the final number is not enough. A coherent end-to-end method is safer than calculator shopping.

What should I copy from an A/B test calculator?

Copy the inputs and decision rule, not only the result: metric, baseline, MDE, alpha, power, tail, method, allocation, exposure, target sample, stopping rule, final counts, SRM, effect interval, p-value, practical threshold, and decision.

Sources

• American Statistical Association statement on p-values
• NIST comparison of two proportions
• Evan Miller on optional stopping
• Microsoft Research on sample ratio mismatch
• Online Controlled Experiments and A/B Tests encyclopedia
• A/B test calculator audit dataset