Atticus Li

I gave the same A/B test inputs to five popular sample-size calculators this week. They returned five different numbers — small differences in some cases, large differences in others. None of them is wrong. All of them are using accepted statistical methods. They just make different assumptions abou

The first time I had to plan an A/B test, I opened a sample-size calculator and stared at five input fields I didn't understand. Baseline conversion rate, OK. Statistical confidence, sure, 95%. But "minimum detectable effect"? "Statistical power"? "Number of variants including control"? I clicked ar

A CRO analyst on my team noticed last week that our pre-test calculator was returning slightly different numbers than the Speero calculator he used to use. Same inputs. Different MDE estimates. The difference was small — about 0.4 percentage points — but it was enough to make him question whether ou

If your A/B test sample-size calculator gives you a number 1.5–2.5% larger than another calculator using the same inputs, the most likely reason is a continuity correction. Specifically, the Fleiss continuity correction layered on top of the standard Casagrande-Pike sample-size formula.

If you have spent any time comparing A/B test sample size calculators, you have probably noticed that they disagree with each other for the same inputs. The reason is not that one of them is broken. The reason is that there are four different statistical formulas in widespread use for the same quest

A CRO analyst on my team flagged a problem this week: she ran the same A/B test inputs through three different sample size calculators and got three different answers. Not slightly different. Visibly different — and visible enough that the team started arguing about which tool was "right."

When a stakeholder says they don't see the variant after a 100% rollout, the cause is almost never the deploy. The 9-item debugging checklist most experimentation teams reinvent painfully — anchored on dataLayer and audience-condition checks, with the recon-team operating model that produces this bug class.

The first test's job is rarely to win — it's to identify the next test. A 3-iteration homepage arc on how to back out of a confounded experiment, isolate the right variable, and turn iterative tests into a causal chain instead of a portfolio of unrelated shots.

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

_By Atticus Li -- Applied Experimentation Lead at NRG Energy (Fortune 150). Creator of the PRISM Method. Learn more at atticusli.com._

Everything we learned from running a multi-brand enterprise testing program: the patterns that win, the mistakes that cost months, and the frameworks that survived our own self-audit.

Marketing analytics teaches reporting. CRO requires hypothesis design, statistical thinking, and behavioral reasoning. Here is the real skills gap between the two roles — and how to close it in 3-6 months.

Remote CRO jobs dominate the 2026 market. Learn which industries are hiring most (SaaS, fintech, energy), salary ranges by level, and the 4 skills that differentiate remote CRO candidates from the competition.

Three tests deployed as 100% personalizations with no holdout groups can never prove ROI. The framework: use A/B testing to learn, AI personalization to scale what works.

Late-funnel tests outperformed early-funnel tests in every comparison. A post-enrollment redesign tripled the primary metric. Here's the commitment escalation principle explained.

Most CRO case studies fail because they lack behavioral mechanism and hide failures. This 6-part template — business context, hypothesis, design, results, impact, learning — shows exactly how to present A/B test work that gets you hired.

Our biggest test win wasn't on the homepage or checkout. It was the confirmation page: more than triple deposit completion. Here's why post-enrollment is your highest-leverage surface.

Not all friction kills conversion. Learn the 6 friction types from real test data — and why removing commitment friction actually backfired in our experiments.

Deploying personalization without a holdout group guarantees you can never prove ROI. Learn why even a 5% holdout enables causal measurement — and how to build the habit before your next rollout.

"Test one thing at a time" is wrong. Our biggest winners changed 5+ things. The real rule: all changes must serve ONE behavioral mechanism. Here's the framework.

We claimed dozens of test winners. The real number was a fraction of that. Here's the honest audit that exposed inflated win rates, impossible data, and circular frameworks.

Anchoring theory predicted that showing lower price comparisons would lift conversion. It didn't. Here's why anchoring fails on experienced decisions — and what the research actually supports.

CTA button copy changes produce near-zero lift — Cohen's h < 0.01 across multiple tests. Here's what actually drives conversions: placement, visual hierarchy, and user readiness.

"Reducing cognitive load" was cited in 8 failed tests, winning just 13% of the time. Friction removal won 64%. Here's the behavioral science behind the difference.

"FREE" won a dramatic positive lift for new customers but lost a significant decline for existing customers. Same product. Same word. Here's why context defeats copy.

GA4 certification teaches reporting. CRO jobs need analysis. Learn the 5 specific GA4 skills hiring managers actually test for: custom funnels, test segments, event tracking, exploration reports, and platform integrations.

The highest-impact CRO tests consistently involve removing something — fields, clicks, visual weight. Adding more rarely wins. Here's the pattern and why subtraction is your best conversion lever.

The highest-impact CRO tests consistently involve removing something — fields, clicks, visual weight. Adding more rarely wins. Here's the pattern and why subtraction is your best conversion lever.

AI found that friction removal won at far higher rates than cognitive load reduction — a pattern humans missed. Here's how AI changes what we test, how we measure, and when we ship.

"Recommended Plans" failed in every test. Users preferred seeing all options. Here's why the Paradox of Choice doesn't apply to high-consideration purchases.

We ran identical tests across multiple brands. Phone CTAs transferred. Recommended plans didn't. Form chunking failed everywhere. Credit check language varied by brand. Here's the framework for knowing which.

Exit modals, recommended plans, and "FREE" retention messaging all backfired in our tests. Reactance theory explains why pushy UX makes users do the exact opposite of what you want.

AI analyzes your entire test history to generate hypotheses grounded in YOUR data — not generic best practices. Here's where AI excels and where humans must lead.

Kahneman's peak-end rule says memory is shaped by the peak and the end. In enrollment funnels, the end is your confirmation page — which explains why it's a better retention investment than your landing page.

Session replays reveal CRO gold — but watching them is unsustainable at scale. AI can now summarize patterns, flag friction, and surface behavioral clusters across thousands of sessions.

Most teams think they're at Stage 3. They're at Stage 1. Here's the 5-stage experimentation maturity model — and the honest diagnosis for where your program actually sits.

Pages converting at 85-90% have a ceiling effect. Copy tweaks produce 0.3% lift (undetectable). Structural redesigns can still win. Here's the framework for optimizing high-baseline pages.

Risk-reduction messaging at the browsing stage, urgency at exploration — both fail. The right intervention at the wrong funnel stage consistently produces flat or negative results. Here's the framework.

A manual meta-analysis of our testing program took weeks of spreadsheet work. AI can do the same audit in hours. Here's exactly what it covers and what it finds.

CTA button copy changes produce near-zero lift — Cohen's h < 0.01 across multiple tests. Here's what actually drives conversions: placement, visual hierarchy, and user readiness.

Every v1→v2 iteration in our enterprise program improved outcomes. Not most — all. Here's the framework for turning failed tests into your most valuable research.

In enterprises, every device split told a different story than the aggregate. Form chunking: desktop showed strong double-digit gains while mobile declined. Here's what your device split is hiding.

We ran identical tests across multiple brands. Credit check copy: a mid-single-digit lift on one brand, a nearly four percent decline on another. Here's why — and the playbook for knowing which concepts transfer.

7 of enterprises had false starts — bugs and errors caught post-launch. Each cost 1-4 weeks. Here's the pre-launch QC checklist that prevents 90% of them.

6 tests had >95% Bayesian probability but didn't reach frequentist significance. One was shipped and won. Here's the practical guide to choosing your statistical method.

We tested "Recommended Plans" 5 times across multiple brands. Every test failed. One lost a double-digit decline. Here's what this taught us about user choice in high-consideration purchases.

enterprises, multiple brands, one high-consideration funnel. Here are the 6 patterns that consistently win — and the 6 that consistently lose. Transferable to any complex purchase.

Nudging failed 5 times. Default bias won. Framing produced opposite results by audience. Here's what dozens of A/B tests reveal about choice architecture in practice.

40% of our tests chose a primary metric too far from the change. Here's the one rule that fixes metric selection — and the framework to apply it to any test.

We tested phone CTAs across 3 brands. Every test passed non-inferiority on digital AND generated incremental sales. Here's the playbook for testing additive channels.

the vast majority of our tests were underpowered — burning traffic that could have powered better experiments. Here's the economics framework for maximizing your testing program's ROI.

After auditing dozens of enterprise A/B tests, I built a scoring rubric — then discovered it was flawed. Here's the honest checklist that actually predicts test outcomes.

Form chunking failed on all multiple brands. Field removal won +12%. Time on form increased, not decreased. Here's what 7 tests taught us about form design.

Engaging homepage copy that users loved reading killed conversion by a double-digit decline. Boring copy that users ignored converted better. Here's the content hierarchy principle that explains why.

"No Hidden Fees" messaging hurt conversion by a modest decline. "FREE" lost a significant decline for existing customers. Here's why transparency backfires when the experience can't deliver on the promise.

4 tests proved their value by NOT hurting the primary metric while generating secondary wins. Here is the complete guide to non-inferiority A/B testing.

Two tests on different brands independently discovered the same UX problem: address search friction. Neither test was designed to find it. Here's how qualitative CRO research works.

Our biggest winner (more than triple) had SRM on desktop. We saved the test by detecting it early and segmenting by device. Here is the practical guide to SRM detection every CRO practitioner needs.

the vast majority of our tests were underpowered — coin-flip chance of detecting real effects. One needed nearly a year. Here's the pre-test calculation that prevents this waste.

Moving a module higher on the page reduced engagement. Making a button a text link shifted 8% of behavior. Here's how visual hierarchy determines A/B test outcomes.

The highest-ICE test in our program produced zero provable ROI. The lowest-ICE test tripled the primary metric. Here's the quarterly roadmap framework that explains why.

"Recommended Plans" failed in every test. Users preferred seeing all options. Here's why the Paradox of Choice doesn't apply to high-consideration purchases — and what does.

Without a centralized test knowledge base, CRO teams repeat failed experiments and lose institutional learning. Here's what happens — and how to fix it.

"FREE" won for prospects but lost for existing customers. Trust isn't one thing — it's two different psychological states. Here's the trust asymmetry framework for CRO.

CUPED uses each visitor's pre-experiment behavior to filter out noise from your A/B test results — often doubling statistical power without adding a single visitor. Here is how it works and when to use it.

AI classification of behavioral mechanisms across our test portfolio revealed that friction removal won at dramatically higher rates than other mechanisms. Humans had missed it entirely.

When we audited our A/B test database, stored p-values disagreed with recomputed values in a significant share of tests. Here's why that happens — and how to prevent it.

Our biggest winner had the lowest prioritization score. Honestly reporting an inflated winner count was actually a fraction of that increased organizational trust. Here's how to build a data-honest testing culture.

One test had the wrong primary metric. Another had two pages sharing one analytics name. A third had impossible data. Here's why analytics validation must be a launch gate.

Five teams, zero coordination. Tests launched without QC. Metrics set up wrong. Experiments contaminating each other. Here's the experiment brief system that fixed it.

The MDE-by-week table shows exactly how detectable your test will be at 1, 2, 4, 8 weeks — the single artifact that prevents underpowered experiments before a line of code is written.

A guardrail caught a nearly six percent decline decline that the primary metric missed. 12 tests had no guardrails at all. Here is the complete guide to guardrail metric selection.

We wasted 2 months on a test that should have been a Go-Do. A very-low-traffic/day page needed nearly a year to reach significance. Here's the framework for knowing when NOT to A/B test.

The most valuable finding from our tests was never the primary metric. It was what session replays revealed about WHY the numbers looked that way. Here's how qualitative transforms quantitative.

The test with our highest ICE score couldn't prove ROI. The lowest-scored test tripled the primary metric. Here's why prioritization frameworks fail at predicting A/B test impact.

Loss aversion is the most-cited principle in CRO and arguably the least useful. Here's why urgency messaging is mostly untestable, and what actually drives conversions.

We recomputed every statistic in our testing program. Stored outcomes disagreed with reality in a third of tests. Here's the step-by-step meta-analysis protocol — and what you'll probably find.

E-commerce CRO doesn't transfer to utilities, insurance, or telecom. Here's the complete playbook for high-consideration industries — from plan selection to post-enrollment activation.

Moving social proof higher on the page didn't help — and in one test, moving reviews above the CTA reduced conversion. Social proof confirms decisions, it doesn't initiate them. Here's the evidence.

Most tests never reach significance. Some should ship anyway. Others shouldn't. Here's the 4-factor framework for the ship-or-kill decision when stats aren't conclusive.

Our biggest wins weren't optimization tweaks — they were product redesigns measured as tests. Here's how to run your testing program like a product team.

Most tests are underpowered before they start. A pre-test feasibility check — MDE by week, runtime flags, traffic thresholds — prevents months of wasted experiment time.

The default effect is the strongest behavioral principle in digital product design and the most consistently ignored. Here's how visual defaults and choice architecture win where copy changes fail.

Users who've invested effort in a multi-step enrollment feel ownership of their progress. Late-funnel interventions outperform early-funnel ones because of this. Here's the behavioral science behind why.

When we measured Cohen's h across our full test portfolio, over 90% of tests produced negligible effect sizes. Here is what that reveals about which tests are worth running — and which never had a chance.

Only a fraction of our tests produced stat-sig wins. Tests scoring ≥7 on design quality won 85% of the time. Here's why running fewer, better tests beats a high test count.

"Is it significant?" is the wrong question for business decisions. Expected loss — how much value you risk by shipping — gives you what p-values never can: a direct answer to the decision you actually face.

---

---

---

---

---

---

80% of A/B tests fail before launch because the hypothesis is wrong. Learn the 4-question framework and EBITDA formula from a $30M experimentation program.

Most A/B test content is binary: winner or loser. Practitioners face 6 distinct outcomes. Learn the decision framework for each from a $30M experimentation program.

Learn how to build an experiment tracking system that compounds learnings into revenue. Includes the Experiment Tracking Maturity Model, real data from 97+ tests, and a step-by-step implementation guide.

Build a CRO test library that compounds growth. This 5-level maturity model framework shows how to organize, tag, and retrieve every A/B test result so your team stops re-running failed experiments.

Compare full-stack, feature flag, analytics-native, and repository-first experimentation platforms. Real experiment data from 100+ tests/year and $30M+ revenue impact.

Learn the proven 5-Layer Organization System to organize AB tests at scale. Real experiment data from 100+ tests/year shows how an AB test organizer eliminates duplicate tests and compounds learnings.

Data study of 97 A/B tests reveals only 27% produce winners. Learn how to analyze AB test results using the Results Interpretation Matrix framework, with real win rates by category.

Compare AB test statistical validation tools to ensure accurate results. Discover best practices and frameworks for effective experimentation.

Explore top CRO program management tools with our comparison guide to boost your conversion rate optimization efforts.

Explore the differences between experimentation knowledge bases. Discover insights from 100+ experiments and learn how to optimize your CRO strategy.

Discover the pros and cons of A/B test repositories and libraries. Learn which approach best suits your experimentation needs.

Explore A/B test repository architecture including schema design, tagging systems, and efficient retrieval methods for optimized experiment management.

A/B test repository best practices help teams preserve experiment knowledge and prevent data loss during staffing changes. Maintain institutional memory effectively.

A/B testing documentation framework provides essential templates and metadata standards to streamline experiment tracking and enhance testing reliability.

Discover why anchoring bias undermines your A/B testing on pricing pages and learn proven strategies to optimize conversion rates effectively.

Discover the best A/B test library software with comprehensive feature comparisons, trade-offs, and key evaluation criteria to optimize your testing strategy.

Build a structured experimentation knowledge base to accelerate growth testing cycles and enable data-driven decision making across high-velocity teams.

Struggling to keep your A/B testing data organized and accessible as your experiments scale? Centralized databases can address this by bringing together user be

Master conversion rate optimization at scale with proven strategies for managing 100+ concurrent A/B tests efficiently. Learn expert techniques to boost results.

Struggling with lost insights from past A/B tests or experiments? Keeping valuable knowledge preserved is crucial to avoiding repeated mistakes. This post will

Experiment tracking systems help product teams measure results and optimize features effectively. Learn best practices for building scalable infrastructure.

Explore experimentation governance best practices for managing SRM, false positives, and bias in your testing programs. Ensure reliable results.

Teams running frequent A/B tests often face disarray without a well-organized experimentation process. Nearly half of organizations lack formal structures, whic

Gap analysis in experimentation reveals untested opportunities that drive growth. Learn to identify, prioritize, and close testing gaps systematically.

CRO teams often struggle to keep valuable insights from slipping through the cracks. Institutional knowledge, or the collective wisdom built over time, is a gam

Managing A/B tests for multiple clients can feel overwhelming. Each project requires precision to ensure data quality and meaningful results. This article will

High-volume testing teams compound growth by systematizing experimentation and learning. Discover how to build scalable testing infrastructure.

Discover how product and CRO teams can effectively share A/B test learnings beyond Slack threads for better collaboration and insights.

Identify errors by analyzing past data patterns, differentiating faulty execution from weak hypotheses to refine future testing methods.

Build a resilient test library that outlasts team changes with maintainable code, clear documentation, and strategic patterns for long-term success.

A/B testing drives better decisions for SaaS teams. For low-traffic products, calculating the right sample size is essential to ensure valid results.

Learn to design a scalable A/B testing repository that efficiently manages over 100 experiments. Master best practices for organization and growth.

Prevent institutional knowledge loss in your A/B testing program by implementing documentation practices and team training. Protect your testing assets today.

Learn how to run meta-analysis on historical A/B test data to uncover powerful insights and improve your testing strategy effectively.

Use your test history to identify winning experiment patterns and predict future success. Learn data-driven strategies for better results.

Discover how loss aversion impacts CTA testing and conversion rates. Learn behavioral economics principles to optimize your calls-to-action effectively.

Discover how meta-analysis reveals hidden patterns across multiple A/B tests. Learn statistical methods to synthesize 50+ experiments and boost conversion rates.

Streamline experiment management across teams with strategies to break down silos between product, marketing, and UX departments effectively.

Discover essential post-test analysis steps to maximize your A/B test results and implement winning strategies effectively.

Master pre-test and post-test calculators to ensure statistical reliability in your research and experiments with proven methods.

Learn how to use a pre-test calculator to determine if your A/B test has sufficient traffic before launching. Ensure statistical validity.

Running A/B tests often takes time, yet many teams fail to review their past experiments for insights. Studies show that more than 90% of tests do not produce s

Conversion rate optimization (CRO) teams often rely on data to make decisions. Statistical significance and Bayesian probability are two popular models in A/B t

Master A/B test documentation with this proven 5-part template that ensures critical insights never slip through the cracks during your optimization work.

Summary: This guide explains how to convert scattered A/B testing insights into a structured, searchable, and reusable repository. It focuses on building instit

The compound effect of experimentation builds momentum over time. Learn why consistent testing in month 12 dramatically outperforms initial month 1 efforts.

Master solo CRO testing with our comprehensive guide on organizing experiments efficiently without dedicated team support.

Scattered A/B test results waste time and money. Learn how consolidating data from Jira, Notion, and spreadsheets improves decision-making and efficiency.

Discover how analyzing 100 experiments reveals strategic patterns that transform data into actionable business insights. Learn proven techniques today.

Learn what happens to your test history when key team members leave and how to protect your QA documentation and data continuity.

Discover what a test repository is and why it's the missing layer in your experimentation stack. Learn how to log A/B test results effectively.

Discover what Sample Ratio Mismatch is and how SRM errors invalidate your A/B tests. Learn to detect and fix SRM issues today.

Learn when spreadsheets become inefficient for experimentation and why upgrading to a dedicated platform improves testing accuracy and results.

Checkout flow tests drive higher conversion rates than homepage tests. Discover data-backed insights from 1,000+ experiments on optimization priorities.

Fix your CRO team's testing cycle. Learn why failed experiments repeat and discover proven strategies to improve test documentation and organizational learning.