Loser
Low confidence
Walmart: Site-wide — Big Navigation A/B Test With 8 Confounded UI Changes
Walmart ran a larger redesign experiment of their navigation. If I observed correctly after triple checking, it was a leap variation with at least 8 changes grouped together. Unfortunately, as of this month it seems that the experiment has stopped with no sight of the variation - hinting at a rejection for whatever reason. In retrospect, I do have some ideas along with my personal and mixed bets on these UI changes. All in all, I suspect a possible confounding situation with some negative changes cancelling out the positive.
Hypothesis
If we test a similar change on our any pages as Walmart rejected, we should be cautious — their rejection suggests it underperformed.
Why This Works
Key Learnings
Viewed 8+ times
Related Entries
Research Finding
Principle
4+ Variation Tests Win 2.4x More Often With 27% Higher Lift
testing_strategyany
Expected lift: 10%-30%
High confidence
Test Idea
Winner
Multiple Steps
navigationlisting
Expected lift: 5%-5%
High confidence
Test Idea
Winner
Pattern #14: Exposed Menu Options
navigationproduct
Expected lift: 0%-5%
High confidence
Research Finding
Principle
Personalized Experiences Generate 41% Higher Impact Than Generic
personalizationany
Expected lift: 20%-50%
High confidence
Best Practice
Loser
Ghost / Outline Buttons Lose to Filled Buttons — Consistently
ctaany
Expected lift: -25%-0%
High confidence