General: Test Quality Peaks at 1–10 Annual Tests Per Engineer; Drops 87% After 30
Hypothesis
High test velocity without prioritization dilutes quality and reduces average lift per experiment.
Test Results
Key Learning
Principle: Use a prioritization framework (PIE, ICE, or custom scoring) before building a test. Quality hypothesis generation matters more than raw test velocity.
When to apply: Integrate this into your experimentation process — it changes how you should prioritize and structure future tests.
How to Apply This to Your Site
This experiment tested general: test quality peaks at 1–10 annual tests per engineer; drops 87% after 30 but produced no statistically significant change. The test was run on a landing page page in the cross-industry industry. Inconclusive results suggest this particular change may not be a priority — focus testing effort on higher-impact areas.
Before you test: Consider that testing strategy tests typically require adequate traffic to reach statistical significance. Run your test for at least 2 full business cycles to account for weekly traffic patterns.
This result reached 95% statistical confidence, meaning there is a very low probability the observed effect was due to chance. Results at this confidence level are generally considered reliable for making business decisions.
What Was Tested
Impact per test peaks when teams run 1–10 tests per engineer annually. Beyond 30 tests per engineer, expected impact drops 87%. This suggests testing more without a prioritization framework destroys value.
Methodology
Build On These Learnings
Save your own experiments, spot winning patterns across your test history, and stop repeating what's already been tried.
Related Experiments
Multiple: Thortful Greeting Cards: Experimentation Program Delivers 30x ROI
Context: Friction during the multiple process causes users to abandon right when they're closest to converting.
General: Testing Strategy Optimization
Principle: Simple A/B limits the solution space. Exploring 4+ design directions dramatically increases the chance of finding a meaningful improvement. Use multivariate or multi-arm testing when prioritization bandwidth allows.
Landing: A/B Testing Lead Gen Page
Problem: How "A/b testing lead gen page" is implemented on the landing can meaningfully affect conversion — this element is worth testing.
Homepage: Meta: Top Homepage CRO Patterns
Principle: The highest-ROI tests on homepages are usually structural (CTA placement, sticky nav, multi-step forms) rather than content changes. Copy matters most for nav labels and CTAs. Social proof works for social platforms but can backfire for professional services. Carousels consistently underperform static alternatives.