P-hacking occurs when researchers try many analyses, exclusions, outcomes or stopping points and highlight the combinations that produce an attractive result. Outcome switching occurs when a study reports a different primary outcome from the one planned, especially if the planned result was disappointing. Neither term proves fraud: choices can arise from uncertainty, curiosity or poor documentation. The problem is that a result discovered after many attempts is presented as though it came from one pre-specified test. Chance patterns then look more convincing than they are. Prospective registration, time-stamped analysis plans, complete outcome reporting and clearly labelled exploratory work help readers distinguish planned confirmation from later discovery. These safeguards matter because selective reporting can distort reviews, policy and treatment decisions even when every published number was calculated correctly.
Key takeaways
- P-hacking exploits flexibility in data collection or analysis.
- Outcome switching changes or omits pre-specified outcomes after results are known.
- A valid calculation can still support a misleading research story.
- Multiple testing increases the chance of apparently significant findings.
- Exploratory analysis is valuable when labelled honestly.
- Registration, protocols, code and full results improve accountability.
The difference between error and distortion
Random error is ordinary variation in samples. P-hacking changes the selection process: many possible results are generated but only favourable ones are emphasised. The reported p-value then fails to reflect the full search. Outcome switching changes which question appears central. Both practices can exaggerate evidence without altering the arithmetic of the selected analysis.
The garden of forking paths
Researchers face choices about variables, time windows, transformations, covariates, subgroups and exclusions. Even without consciously chasing significance, looking at the data can influence those choices. Each path may seem defensible alone. Together they create many possible analyses, increasing the chance that one produces a striking result by accident.
How repeated testing changes probability
If many independent null hypotheses are tested at a five per cent threshold, at least one false positive becomes increasingly likely. Real analyses are often correlated, so a simple multiplication does not give the exact risk. The core lesson remains: a p-value from the winning test cannot be interpreted as though it were the only test attempted.
Optional stopping
A study may be checked repeatedly and stopped when significance appears, or extended when a result narrowly misses the threshold. Standard tests assume a sampling plan; unplanned peeking changes their behaviour. Sequential designs can be legitimate when their decision rules and error control are planned in advance. The problem is undisclosed flexibility, not every early look at data.
Flexible exclusions and subgroups
Removing an “outlier” can be justified when a measurement failed, but trying several exclusion rules and reporting only the helpful one is selective. The same applies to age bands, regions or other subgroups. Subgroup patterns based on small numbers are especially unstable. Reports should state all planned groups and label additional analyses as exploratory.
What outcome switching looks like
A trial may register pain reduction as its primary outcome but publish a secondary laboratory marker as the main success. The laboratory result may be real, yet readers lose the context that the planned clinical outcome was not supported. Switching can also involve changing the measurement time, scale or definition after results are available.
Why primary outcomes matter
The primary outcome focuses sample-size planning and limits the number of main claims. It should represent the central question rather than the easiest variable to move. Secondary outcomes can add important evidence, but they usually receive less statistical protection. A clear hierarchy prevents a long list of measures from becoming many opportunities to declare victory.
HARKing and post-hoc stories
HARKing means hypothesising after results are known while presenting the hypothesis as prior. Scientific explanation often develops after seeing data, so post-hoc hypotheses are not worthless. They become misleading when discovery is rewritten as prediction. A new hypothesis should be tested with new data or presented as a tentative explanation requiring confirmation.
Exploration is not the enemy
Exploratory analysis finds unexpected patterns, checks assumptions and generates questions. Banning it would weaken science. Researchers can protect its value by showing the range of analyses tried, sharing code and distinguishing exploratory from confirmatory claims. A transparent surprise is more useful than a supposedly predicted result whose history has been concealed.
Preregistration and trial registration
Preregistration time-stamps questions, outcomes and analysis decisions before results are known. Clinical-trial registries also let readers compare publications with original plans. WHO describes prospective trial registration as a scientific and ethical responsibility because selective reporting leaves an incomplete evidence base. Registration is useful only when records are specific, accessible and kept up to date.
Protocols and statistical analysis plans
A detailed protocol explains design and outcomes; a statistical analysis plan specifies transformations, models, exclusions and missing-data handling. Amendments can be legitimate when circumstances change. They should be dated, justified and distinguished from decisions made after unblinding. Transparency allows readers to judge whether a change was methodological improvement or result-driven selection.
Registered Reports
In a Registered Report, a journal evaluates the question and method before results exist and may offer in-principle acceptance. This reduces incentives to favour a dramatic outcome. It does not guarantee a perfect design or eliminate analytical judgement, but it shifts attention from whether the result is positive to whether the proposed test is informative.
Full results and outcome tables
Publishing every pre-specified outcome, including null and adverse results, prevents one success from hiding many failures. Registry summary results can make evidence available even when a journal article is delayed. WHO guidance stresses timely and complete disclosure because unreported studies and outcomes can mislead systematic reviews and waste participant contributions.
How readers can check a study
Look for a registration number, compare outcome names and time points, check whether enrolment and exclusions match the plan and note how many analyses appear. Search for protocols, supplements and registry results. A discrepancy is not automatic misconduct; authors may provide a sound explanation. The key is whether changes are visible and justified.
Robustness and multiverse analysis
Analysts can show whether conclusions survive several reasonable specifications rather than presenting one preferred model. Sensitivity analyses should be planned where possible and reported comprehensively. If the conclusion appears only under one narrow choice, that fragility matters. Robustness cannot rescue a biased sample, poor measurement or an outcome invented after the fact.
A classroom demonstration
Students can simulate twenty fair-coin experiments and record the most extreme result. Reporting only that experiment makes chance look mysterious; displaying all twenty restores context. They can then write separate “planned” and “exploratory” conclusions. The exercise shows why transparency about the number of opportunities is essential to interpreting statistical evidence.
Distinguishing planned from unplanned multiplicity
A study can contain several outcomes and analyses without being p-hacked. The risk depends on whether the family of tests and error-control method were defined in advance and whether all results are visible. Researchers may adjust significance thresholds, use hierarchical testing or control false discovery rates. These methods do not eliminate judgement, but they align the interpretation with the number of opportunities. Readers should be suspicious of hidden multiplicity, not of every paper that responsibly reports more than one question.
The role of incentives
Researchers work in systems that reward novel positive findings, quick publication and simple stories. Journals, funders, employers and media can therefore encourage selective emphasis even without explicit misconduct. Reform must address incentives as well as individual behaviour: value null results, replication, data curation and careful methods; provide time for complete reporting; and evaluate research quality rather than counting striking papers. Training alone cannot solve a problem when career structures continue to reward the opposite choices.
Corrections without stigma
A transparent paper may disclose that an outcome changed because an instrument failed or a protocol became unsafe. Treating every amendment as scandal encourages concealment. Registries and journals should make changes easy to record while preserving the original plan. The important distinction is whether the change occurred before or after relevant results were known, whether it was justified and whether all planned outcomes remain reported. Correction and amendment are signs of a functioning record when the history is visible.
Evaluating a body of evidence
One study rarely settles a question. Systematic reviews should search registries, reports and unpublished results, compare protocols with papers and assess selective reporting. Replications using fresh data reduce the advantage of a chance pattern found through flexible analysis. Convergence across methods matters more than a stack of nearly identical significant results. When effects shrink in larger or preregistered studies, the earlier literature may have overestimated them even if no single research team acted dishonestly.
Why data sharing is not enough
Open data allow reanalysis, but they do not automatically reveal every abandoned model, unreported outcome or decision made after looking at results. A complete transparency package links data with protocol, analysis plan, code, amendments and a clear outcome table. Privacy or commercial limits may prevent public release of individual records, especially in health research; controlled access and auditable analysis can still improve accountability. Openness is most useful when it reconstructs the sequence of decisions, not when it simply deposits an unexplained spreadsheet.
A fair way to discuss suspected bias
Criticism should compare documents and analyses rather than infer intent from an unwanted result. State the discrepancy, show how it might affect interpretation and invite explanation. Some changes correct errors or respond to events outside researchers’ control. Others increase the number of unreported opportunities and weaken evidence. Separating effect from motive keeps the discussion scientific. The practical question is whether a reader has enough information to reproduce the planned test, see deviations and judge the full set of outcomes.
Questions to ask before accepting the conclusion
For any claim about p-hacking and outcome switching, identify what was measured directly, which steps relied on models or judgement, and whether the sampled places and times represent the wider question. Check the units, comparison group, missing observations and uncertainty range. Look for an independent method with different weaknesses and ask whether it supports the same central result. Finally, separate a useful screening signal from evidence strong enough for an expensive or irreversible decision. This checklist does not demand perfect certainty; it ensures that the confidence of the conclusion matches the quality and scope of the evidence.
Related Earth Site reading
Useful background is available in publication bias, preregistration and Registered Reports, interpreting p-values and robustness and replication. These public Earth Site pages were checked as internal destinations; the independent sources listed in the companion notes support the research in this article.
Conclusion
P-hacking and outcome switching exploit the gap between the many analyses that could be performed and the small selection readers eventually see. They can produce a persuasive but incomplete account even when the published calculation is technically correct. Clear primary outcomes, prospective plans, dated amendments, full reporting and accessible code make that selection process visible. Exploration should continue, but it should be labelled and confirmed independently. The goal is not to remove human judgement from research; it is to prevent hidden flexibility from masquerading as strong prior evidence.
Frequently asked questions
Is every change from a preregistered plan wrong?
No. Changes can be necessary, but they should be dated, justified and reported transparently.
Is p-hacking the same as fabrication?
No. P-hacking selects among analyses; fabrication invents data. Both can mislead, but they are different problems.
Can exploratory findings be published?
Yes. They should be labelled exploratory and tested in new data where possible.
Does a significant p-value prove p-hacking?
No. Evidence comes from discrepancies, undisclosed flexibility, implausible patterns and the broader research record.
How does trial registration help?
It preserves a public record of planned outcomes and design so later reports can be compared with the original intentions.
