GrowthLayer methodology
How GrowthLayer grades experiment evidence
Read the tiers as a decision ladder, not as school grades. A higher tier supports a stronger business use. It does not guarantee that the same change will work for every company, audience, or market.
Methodology version: growthlayer-evidence-v2 · Updated July 21, 2026
The 10-second version
The action is the maximum responsible use of the evidence. Business impact, rollout cost, reversibility, and how closely your context matches the test still affect the final decision.
The four judgments shown on a record
Evidence tier
What decision this record can responsibly support. A failed quality gate always overrides a high completeness score.
Record completeness
How much required information is present. A complete record is easier to inspect, but complete does not automatically mean correct.
Integrity
Whether a known assignment, metric, tracking, drift, or contamination issue could bias or invalidate the result.
Research support
Why the idea was worth testing and which customer problem it addressed. This is useful context, but it answers a different question from causal impact.
What each tier means for a decision
Each tier is a hard gate. A polished write-up cannot compensate for broken data, and a large reported lift cannot compensate for a test that cannot be checked.
| Tier | What you can trust | Responsible business use |
|---|---|---|
| E0 · Compromised | A serious data, metric, allocation, or implementation problem means the result cannot show what caused the outcome. | Do not use for a business decision. Keep only as a QA or process lesson. |
| E1 · Reported only | An outcome is reported, but too little of the setup or data is available to check whether it is reliable. | Use for inspiration or follow-up, not as proof. |
| E2 · Directional | The test idea, experience, metric, sample, duration, and result are documented, but key data or analysis details are missing. | Use to prioritize a local test, not to justify a rollout by itself. |
| E3 · Reviewable | Arm-level results, expected traffic allocation, the analysis method, and an SRM check are available, with no known critical integrity issue. | Use for cautious, context-specific decisions. Retest when the cost or risk is high. |
| E4 · Audit-ready | The experiment was planned before launch and includes sample and stopping rules, guardrails, diagnostics, and enough data to reproduce the analysis. | Strong basis for a rollout decision in the tested context. Monitor guardrails after launch. |
E5 · Replicated is not a single-test badge. Multiple sufficiently independent E3 or E4 experiments must support the same conclusion, with explicit inclusion rules, comparable outcomes, heterogeneity review, and sensitivity analysis. It is the best basis for a reusable playbook where the tested contexts match, not a universal guarantee.
Research and experiments answer different questions
Qualitative research helps explain needs, language, usability failures, motivations, and mechanisms. Quantitative observational research measures prevalence, funnels, cohorts, associations, and trends. A controlled experiment estimates the causal effect of an assigned treatment. Mixed-methods work is valuable when the methods are planned around the same research question, but combining methods does not turn an association into causality.
This separation does not make qualitative research less valuable. It prevents two different questions from being confused: research explains why a problem may exist; a controlled experiment estimates whether a specific change caused an outcome and by how much.
Qualitative method tags
Interviews, contextual inquiry, moderated and unmoderated usability tests, diary studies, field observation, focus groups, open-ended surveys, card sorts, tree tests, support or sales-call review, coded session replay, heuristic review, competitive review, and literature review.
Quantitative observational tags
Funnel, cohort, segmentation, event-log, closed-ended survey, heatmap, performance telemetry, financial, and market-benchmark analyses. Each record also captures its unit, window, sample, exclusions, missingness, instrumentation, and confounders when disclosed.
What the experiment audit checks
Before exposure
Problem evidence, falsifiable hypothesis, audience, randomization unit, control and variants, one primary metric, secondary and guardrail metrics, baseline, MDE, alpha and power or Bayesian decision thresholds, sample plan, traffic allocation, multiplicity, and stopping rule. A server timestamp distinguishes a real plan from a plan reconstructed after seeing results.
During execution
Actual dates and arm counts, exposure and instrumentation QA, SRM against the declared allocation, missing telemetry, bot or join-rate issues, design drift, variant changes, traffic-source shifts, campaigns and targeting changes, concurrent experiments, outages, pauses, ramps, and restarts.
At analysis
Absolute and relative effects with uncertainty intervals, the declared primary analysis, multiple-comparison handling, practical significance, guardrails, exploratory segments, decision, rollout, and follow-up. A p-value is not treated as the probability that the result is true, consistent with the American Statistical Association statement.
Choose how you will check results before launch
Most web teams only need to make one simple choice. GrowthLayer saves that choice and applies the same rule at the end, rather than running several methods and selecting whichever answer looks best.
Check once at the planned end
Best for most ordinary A/B tests. Choose the main metric, smallest useful change, traffic split, and sample or duration before the test starts.
Monitor safely while it runs
Use a monitoring-safe method chosen before launch. An ordinary end-of-test p-value cannot be used to stop whenever the result starts looking favorable. GrowthLayer keeps final totals exploratory when the original sequential path is not available.
- Bayesian test: allowed when the prior, smallest useful effect, acceptable loss, minimum data, maximum duration, monitoring cadence, and stopping rule were saved before results were viewed. Probability alone never creates a winner. GrowthLayer currently reports Bayesian calculations as sensitivity evidence while the full versioned protocol is not captured.
- Historical exact-count test: GrowthLayer can recheck the numbers, but labels the analysis retrospective unless the original plan is attached.
- Missing arm counts: preserve what the source reported and clearly show what cannot be verified. GrowthLayer does not manufacture a new probability, interval, or winner.
- Multiple variants or audience cuts: choose the main comparison first. Device and audience differences remain exploratory unless the difference between those groups is tested directly.
- Before-and-after rollout: treat it as a rollout analysis requiring suitable time-series or comparison data, not as a randomized A/B test.
Show the statistical details
- Fixed-horizon frequentist: define the business MDE, alpha, power, sample, and stopping point first; report an effect and confidence interval with the p-value.
- Sequential frequentist: use an alpha-spending, group-sequential, confidence-sequence, or other anytime-valid method declared before repeated looks. Continuous monitoring breaks ordinary fixed-horizon inference; Johari, Pekelis, and Walsh describe an always-valid approach.
- Bayesian: declare the likelihood, prior, practical-equivalence or minimum-effect region, expected-loss threshold, minimum information, multiplicity treatment, calibration version, and stopping rule. Probability-to-be-best by itself is directional context.
Data quality can overrule a result
GrowthLayer checks SRM before interpreting outcome metrics. Microsoft Research calls SRM a symptom of multiple possible data-quality failures and warns that ignoring its cause can reverse a ship decision. A detected SRM is therefore an investigation gate, not a number to explain away. The same principle applies to wrong metrics, telemetry loss, treatment changes, and unreconciled result mismatches.
Read the Microsoft SRM taxonomy and its metric-interpretation pitfalls.
How meta-analysis eligibility works
E3 and E4 records can enter confirmatory synthesis when their integrity review is clear. E2 and caution-flagged E3 records can support exploratory analysis, but they are never silently pooled with stronger evidence. E0 and E1 records are excluded from causal estimates. GrowthLayer preserves metric definitions, uncertainty, context, dependencies, and study weights rather than averaging published lift percentages.
The Cochrane Handbook is explicit that heterogeneity can make a single average misleading. GrowthLayer therefore treats industry, business model, company-size band, surface, device, audience, metric, and treatment mechanism as moderators, not decorative tags.
What the first full audit found
On July 20, 2026, GrowthLayer applied this policy to 160 evidence-library records. Eighty-seven were E0 or E1, 50 were E2, 23 were E3, and none reached E4. No record qualifies for confirmatory synthesis because none includes both a verifiable pre-exposure registration and stopping rule with a clear integrity review.
The same audit found usable arm-level counts in 119 records, a documented hypothesis in 96, a primary metric in 134, a planned sample in 29, an MDE in 34, and guardrails in 24. The audit also found unresolved SRM and critical integrity issues. Small cells are combined or suppressed in public aggregate reporting so individual records cannot be inferred from the totals.
These are coverage and quality counts, not a performance benchmark. GrowthLayer publishes them because a grading system should expose where its own evidence falls short. Assessments are versioned and rerun when a record gains better data.
What this means when you browse GrowthLayer
Every experiment shows a compact tier near its outcome. Open “Why this rating” to see completeness, integrity, the analysis family, and the most important missing fields. Research supporting the hypothesis appears separately. Unverified claims remain labeled as reported, identifying details are excluded from public summaries, and no result is called replicated because one test looked persuasive.