GrowthLayer meta-research
What 160 experiment records reveal about evidence quality
We set out to find reusable CRO patterns. The first defensible finding was more basic: most experiment records preserve the outcome, but not enough of the decisions needed to verify it.
This is meta-research, not a causal meta-analysis
No experiment currently passes the confirmatory pooling gate, and no group of five reviewable records has established comparable treatments, outcomes, contexts, and independence. We therefore mapped the evidence and audited reporting quality. We did not calculate a pooled lift, win rate, or “best” tactic.
The result in four numbers
The average completeness score was 53.7 out of 100. Completeness measures field coverage, not truth: a complete record can still contain an integrity problem, and an honest sparse record should not be presented as stronger than it is.
The collection remembers outcomes, not decisions
Every record was missing both a timestamped pre-exposure analysis plan and a stopping rule. Guardrails were absent from 85%, planned sample size from 81.9%, and MDE from 78.8%. These omissions make it difficult to distinguish a planned decision from one reconstructed after the result was visible.
Execution history is part of the result
Design drift appeared in 16 records (10%), and a possible mid-test implementation change appeared in 10 (6.3%). These are not administrative footnotes. A changed treatment, audience, campaign, metric, or tracking implementation can change the question the experiment answers.
The practical fix is an append-only execution log: record ramps, pauses, releases, targeting changes, instrumentation changes, concurrent experiments, outages, and metric-definition changes while the test is running.
Coverage is not proof of effectiveness
Product listing pages were the largest named page context, with 44 records (27.5%). Product comparison, pricing, homepage, CTA, mobile, copy, form, and layout were the most visible named categories. Those counts describe what the library contains. They do not show that these areas produce larger effects or win more often.
Industry, business model, and company-size context were unclassified across the experiment corpus. Until those moderators are populated, cross-company transfer claims would be speculation.
Research needs its own quality record
Of 2,483 research records, 1,503 had a classified research mode and 581 had a documented method. Participant count was not documented in the structured participant field. That does not make qualitative research useless; it means claims about saturation, prevalence, or sample breadth cannot be audited from the current record.
Surveys, moderated and unmoderated usability tests, market benchmarks, heatmaps, tree tests, competitive reviews, prior experiments, coded session replays, and card sorts all appear in the taxonomy. They should not be collapsed into one “research-backed” checkbox because each method supports a different inference.
What teams should preserve
- 1. Before launch: problem evidence, hypothesis, primary metric, guardrails, baseline, MDE, sample plan, allocation, and stopping rule.
- 2. During the test: exposure counts, SRM, instrumentation checks, ramps, changes, campaigns, incidents, and concurrent experiments.
- 3. At analysis: effect and interval, declared method, multiplicity handling, guardrails, practical significance, decision, and follow-up.
- 4. For research: named method, recruitment, participant count, tasks or guide, analysis approach, disconfirming evidence, and limitations.
Method and privacy
GrowthLayer used the latest versioned assessment for each experiment record. The library spans different contexts and is not a representative sample of all CRO programs; records were not assumed independent. Public reporting uses disclosure-controlled aggregate fields. Small partition cells are combined with a broader category, and small cross-tab cells are suppressed.
The report is reproducible under methodology version growthlayer-evidence-v2. Read the evidence-tier methodology for the hard gates and statistical rules.
The next analysis should be earned
The next milestone is not a larger record count. It is a set of comparable E4 experiments with explicit inclusion rules and enough context to test a narrow mechanism. Only then should a protocol choose fixed-effect, random-effects, Bayesian hierarchical, or sequential synthesis.