Listing: Long Titles
Hypothesis
If we implement 'Long Titles' on listing pages (In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines), then key conversion metrics will improve.
Test Results
Key Learning
Context: The headline on the listing may not resonate with what users actually care about or address their top objections.
What was tested: REAL-WORLD TEST: 'Long Titles' was tested on a live listing page. The test involved 68,822 real visitors. Full statistical results require paid access. Test methodology: In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t requi...
Result: No statistically significant difference was detected. Inconclusive copy tests usually mean both versions are equally (in)effective at addressing user motivations. Try a fundamentally different angle.
How to Apply This to Your Site
This experiment tested listing: long titles but produced no statistically significant change. The test was run on a category page page in the cross-industry industry. Inconclusive results suggest this particular change may not be a priority — focus testing effort on higher-impact areas.
Before you test: Consider that copy & messaging tests typically require large sample sizes to detect small effects. Run your test for at least 2 full business cycles to account for weekly traffic patterns.
What Was Tested
In 2012 a Microsoft employee working on Bing had an idea about changing the way the search engine displayed ad headlines. Developing it wouldn’t require much effort—just a few days of an engineer’s time—but it was one of hundreds of ideas proposed, and the program managers deemed it a low priority. So it languished for more than six months, until an engineer, who saw that the cost of writing the code for it would be small, launched a simple online controlled experiment—an A/B test—to assess its impact. Within hours the new headline variation was producing abnormally high revenue, triggering a “too good to be true” alert. HBR, September–October 2017 Issue, https://hbr.org/2017/09/the-surprising-power-of-online-experiments Note : This experiment was a solid success and replicated multiple times over a period of months. It worked at Bing and had a profound influence. The only reason why we atributed a 0.25 point (a "Maybe") was because we don't have the exact sample size and conversion data.
Methodology
Build On These Learnings
Save your own experiments, spot winning patterns across your test history, and stop repeating what's already been tried.
Related Experiments
Content Page: Inline Link Nudge
Context: Form input design on the content page affects completion rates — label placement, validation timing, and field clarity all matter.
Does a 90-Day Plan-Change Guarantee Badge Increase Click-Throughs?
Ambiguity > absence. A vague benefit callout can create more friction than no callout at all: visitor diagnostics showed users were drawn in by the badge (time-on-page up, bounce rate down) but exit rate rose and FAQ-section attractiveness spiked — a signature of users searching for answers and not finding them. The same concept won at a sister brand whose variant used descriptive benefit-framed copy ("we'll help you find the right plan if this isn't a fit"); the variant in this test used short labelled-badge copy that raised more questions than it answered. The lesson is not that benefit guarantees fail — it's that surfacing one with insufficient context can backfire by introducing uncertainty the page doesn't resolve.
Home landing: Empowering Headline
Context: The headline on the home landing may not resonate with what users actually care about or address their top objections.
Product: More For Less Headline
Context: The headline on the product may not resonate with what users actually care about or address their top objections.