Multiverse analysis and specification-curve analysis examine how a research result changes across many reasonable analytical choices. A dataset may permit several defensible definitions, exclusion rules, transformations, covariates and statistical models. Reporting only one path can make a conclusion look more certain than it is. A multiverse runs a planned set of alternative analyses and studies the distribution of estimates. A specification curve usually orders those estimates visually and shows which choices produced each result. These methods are not invitations to include every imaginable model. The set must be scientifically justified, transparently defined and interpreted with dependence between specifications in mind. Their purpose is to reveal whether a finding is robust, conditional on particular decisions or too unstable for a simple headline.
Key takeaways
- Many datasets support several reasonable analytical decisions.
- A multiverse analyses a defined set of defensible combinations.
- A specification curve links effect estimates to the choices behind them.
- The specifications are correlated because they reuse the same data.
- Unreasonable models should not be included merely to enlarge the multiverse.
- Robustness across choices strengthens interpretation but does not prove causation.
Why one analysis may be misleading
Researchers choose how to define variables, handle missing values, exclude cases and control for other factors. Each choice may be plausible, yet the combination can alter the estimate. This flexibility is sometimes called researcher degrees of freedom. It becomes misleading when a preferred result is selected after trying alternatives and the unsuccessful paths remain invisible.
What counts as a reasonable specification
A defensible choice follows the research question, study design and established knowledge. Analysts should state inclusion criteria before seeing which choice gives the desired result. A model that violates measurement or causal assumptions is not rescued by appearing in a large grid. Expert reasoning narrows the multiverse to alternatives that a careful researcher could genuinely defend.
Building the decision grid
The analyst lists choice points, such as two outcome definitions, three covariate sets and two exclusion rules. Their combinations produce twelve specifications, although some combinations may be incompatible. The grid should record the exact rule for each path and the reason for including it. Software code can generate the models consistently and reduce transcription error.
What a specification curve shows
Each estimate is plotted, commonly ordered from smallest to largest, with confidence intervals and a panel identifying the choices used. Readers can see whether estimates share a sign, how large they are and which decisions cluster with stronger or weaker results. The graph is a map of analytical sensitivity, not a contest in which the majority automatically wins.
Effect size before significance
Counting significant results can be misleading because thresholds discard information. Examine the magnitude, direction and uncertainty of estimates. The confidence-interval guide explains why an interval conveys the range of values compatible with data and assumptions more clearly than a binary label.
Dependence between results
Specifications reuse participants and variables, so their estimates are correlated. Treating one thousand specifications as one thousand independent studies would exaggerate precision. Formal tests use resampling, permutation or other procedures designed for the shared data structure. Descriptive displays remain valuable even when no single combined p-value is appropriate.
Multiverse analysis versus sensitivity analysis
A conventional sensitivity analysis may vary one or two assumptions around a primary model. A multiverse is broader and more systematic, often exploring combinations. Both ask how conclusions depend on choices. The term matters less than whether the alternatives are justified, complete enough for the question and reported without cherry-picking.
Preregistration and exploratory use
A multiverse can be preregistered, defining the grid and interpretation before outcomes are inspected. It can also be exploratory, especially when the main purpose is to discover which modelling assumptions matter. The status should be clear. Exploratory results can guide later confirmation but should not be presented as if the entire plan preceded the data.
Sampling problems remain
Running many models does not make a biased sample representative. If the data omit important groups, every specification may reproduce the same limitation. The sampling-bias guide explains why transparent modelling cannot repair a population that was never observed.
Correlation and causal claims
A stable association across many regressions does not by itself establish that one variable causes another. Unmeasured confounding, reverse causation and measurement error may affect every path. The correlation and causation guide provides the wider framework. A multiverse tests analytical robustness, not every alternative causal explanation.
Missing data choices
Complete-case analysis, imputation and model-based approaches rest on different assumptions about why data are missing. Including defensible alternatives can show whether the result depends on one treatment. However, a multiverse should not mix methods whose assumptions contradict known data collection without explanation. Missingness mechanisms deserve their own diagnostics.
Model diagnostics
A specification can be technically fitted yet still be unsuitable because residuals, separation, influential observations or poor convergence invalidate it. Analysts should run diagnostics and flag failed or unstable models rather than silently dropping them. The removal rules must not depend on whether the estimate is attractive.
A worked example
Suppose researchers examine whether urban green space is associated with wellbeing. Reasonable choices include satellite vegetation within 300 or 1,000 metres, two validated wellbeing scales, alternative treatment of income and exclusion of recent movers. The curve shows mostly positive estimates, but the association is small and disappears when neighbourhood deprivation is modelled flexibly. The honest conclusion is conditional: evidence of association exists, but its size depends on social adjustment.
How peer review can use the method
Reviewers can inspect whether the grid omits obvious alternatives or includes implausible ones. They can ask for diagnostics and code rather than request whichever single model they prefer. The peer-review guide explains why this scrutiny can improve work without guaranteeing a correct result.
Communicating the conclusion
A useful summary states the range of estimates, the proportion or pattern with the same direction, the choices that matter most and the substantive importance of the effect. Avoid claiming that a result is “robust to everything”. The analysis only covers the specified choices and the available data; unknown measurement, design and theory problems remain outside the grid.
Specification curves in meta-analysis
Meta-analysis also contains choices: which studies qualify, which effect size is selected, how dependent outcomes are handled and whether a fixed- or random-effects model is used. A specification-curve approach can display results across defensible combinations. This may reveal why two reviews of the same literature reached different conclusions. It also exposes when a headline depends on one inclusion rule or treatment of an outlying study.
The method cannot repair poor primary evidence or publication bias automatically. If every specification draws from studies with similar measurement flaws, the curve may look stable while the underlying construct remains weak. Meta-analytic multiverses should therefore be combined with risk-of-bias assessment, searches for missing results and substantive judgement about whether the studies address a sufficiently common question.
Interpreting a mixed curve
A curve containing positive, near-zero and negative estimates does not mean “anything is possible” without further analysis. Identify which choices divide the results. Perhaps one outcome definition captures short-term behaviour while another measures long-term change, or one covariate is a confounder while another may be a mediator. The pattern can reveal that apparently technical decisions encode different scientific questions.
Researchers should then refine the theory and collect evidence that discriminates between explanations. A multiverse is most useful when it directs attention to the sensitive assumption, not when it becomes a decorative wall of estimates. The final report should say which specifications were preferred before analysis, which alternatives were tested and what future design would reduce the remaining ambiguity.
Conclusion
Multiverse analysis and specification curves make analytical flexibility visible. They show whether a conclusion survives a structured set of reasonable definitions, exclusions and models, and they identify the decisions that drive disagreement. Their credibility depends on a scientifically justified specification set, proper diagnostics and recognition that the results are dependent. Used well, they replace a fragile single-path story with a more informative account of what the data support, under which assumptions and with what remaining uncertainty.
Frequently asked questions
Does a multiverse run every possible model?
No. It should include a justified set of reasonable specifications, not arbitrary or invalid combinations.
Is the most common estimate automatically correct?
No. Frequency across correlated specifications is descriptive and does not establish truth.
Can specification curves prove causation?
No. They test robustness to analytical choices, not all confounding or design problems.
Should non-significant models be hidden?
No. The point is to display the full planned set, including uncertain and contradictory results.
Can the method be preregistered?
Yes. Predefining choices and interpretation reduces the influence of observed outcomes.
