General: Big Navigation A/B Test With 8 Confounded UI Changes
Hypothesis
If we test a similar change on our any pages as rejected, we should be cautious
Test Results
Key Learning
Problem: Users arriving at the general can't efficiently find what they're looking for, increasing bounce rates.
What was tried: rejected this UI change (Nov 9, 2023). Rejection suggests the change underperformed the control
Why it failed: Navigation changes are risky because they disrupt established muscle memory. Test with new visitors separately.
How to Apply This to Your Site
This test showed that general: big navigation a/b test with 8 confounded ui changes hurt conversions. The change was tested on a landing page page in the e-commerce industry. Avoid replicating this exact approach — instead, consider testing the opposite direction or a more subtle variation.
Before you test: Consider that navigation tests typically require adequate traffic to reach statistical significance. Run your test for at least 2 full business cycles to account for weekly traffic patterns.
What Was Tested
ran a larger redesign experiment of their navigation. If I observed correctly after triple checking, it was a leap variation with at least 8 changes grouped together. Unfortunately, as of this month it seems that the experiment has stopped with no sight of the variation - hinting at a rejection for whatever reason. In retrospect, I do have some ideas along with my personal and mixed bets on these UI changes. All in all, I suspect a possible confounding situation with some negative changes cancelling out the positive.
Methodology
Build On These Learnings
Save your own experiments, spot winning patterns across your test history, and stop repeating what's already been tried.
Related Experiments
Listing: Instant Filter Results
Context: Users can't quickly find relevant products or content on the listing, leading to frustration and early exits.
Product: Exposed Menu Options
Problem: How prices are displayed on the product directly influences perceived value and willingness to buy.
Listing: Multiple Steps
Problem: Multi-step processes on the listing can overwhelm users if they can't see how far along they are or how much is left.
Product: Open In A New Tab
Context: Users arriving at the product can't efficiently find what they're looking for, increasing bounce rates.