General: Commonly Used Metrics Are Poor Predictors of Business Impact
Hypothesis
Teams that optimize for CTR or bounce rate may be running tests that don't move revenue.
Test Results
Key Learning
Principle: Always tie test metrics to a revenue or retention outcome. Vanity metrics (CTR, bounce rate) are easy to move but rarely correlate with business results. Use downstream metrics wherever possible.
When to apply: Audit your current general metrics and ensure each one maps to a revenue or retention outcome.
How to Apply This to Your Site
This experiment tested general: commonly used metrics are poor predictors of business impact but produced no statistically significant change. The test was run on a landing page page in the cross-industry industry. Inconclusive results suggest this particular change may not be a priority — focus testing effort on higher-impact areas.
Before you test: Consider that measurement tests typically require adequate traffic to reach statistical significance. Run your test for at least 2 full business cycles to account for weekly traffic patterns.
This result reached 95% statistical confidence, meaning there is a very low probability the observed effect was due to chance. Results at this confidence level are generally considered reliable for making business decisions.
What Was Tested
3 out of the 5 most commonly used test metrics have the lowest correlation with actual business outcomes. Search optimization shows 2.3% expected impact but appears in only 1.3% of tests
Methodology
Build On These Learnings
Save your own experiments, spot winning patterns across your test history, and stop repeating what's already been tried.