Skip to main content

Reported Losing A/B Tests

Each record reports a losing outcome. Study the treatment and context, but do not infer that every related change will fail; inspect the evidence tier and available limitations first.

3
Reported losses
Tiered
Evidence disclosed
Copy & Messaging
Most common category
1
Industries covered

Reported Losses by Category

All 3 Reported Losing Experiments

E1 · Reported onlyreported loser

Does Adding a Shopping CTA to the Main Navigation Drive Plan Views?

Button engagement alone does not show whether the element helped conversion. The topline movement was small and uncertain, while the diagnostic path exposed a larger problem: only about six of every hundred button interactions reached the next funnel stage. The label appears to have attracted people looking for account access rather than shopping, and an added modal inserted another step before the destination. Some activity also shifted away from stronger existing paths. For global navigation changes, evaluate downstream completion by origin page and intent; raw click volume can reward the wrong behavior.

CTAEnergy & UtilitiesSample not disclosedCompleteness 32/100
E1 · Reported onlyreported loser

Does Restructuring Plan Detail Cards Improve Click-Through?

The intervention rearranged the presentation, but the diagnosed problem concerned comprehension. Earlier user feedback pointed to three obstacles: people could not estimate total cost, offer labels did not explain meaningful differences, and the page made simultaneous comparison difficult. Reordering the same unclear information left those obstacles intact. After reaching the planned sample, the observed movement was small and negative. The practical lesson is to make the test variable match the user problem: change the cost explanation when cost is unclear, improve differentiation when offers seem interchangeable, and change layout when navigation or scanning is the issue.

LayoutEnergy & UtilitiesSample not disclosedCompleteness 32/100
E1 · Reported onlyreported loser

Does a 90-Day Plan-Change Guarantee Badge Increase Click-Throughs?

Making the guarantee more visible did not make it more persuasive. The behavioral indicators suggest the badge attracted attention and then created unanswered questions: visitors spent longer on the page, exits increased, and engagement shifted toward the FAQ. A comparable implementation used a fuller explanation, while this version relied on a short label. The practical lesson is to explain eligibility, timing, and fees beside the claim. A guarantee can increase uncertainty when its prominence grows faster than its supporting context.

Copy & MessagingEnergy & UtilitiesSample not disclosedCompleteness 32/100

Also Explore

Diagnose Before Generalizing

Compare the hypothesis, implementation, audience, metric, and integrity notes before deciding whether a reported loss applies to your product.

View Plans & Pricing