A p-value is a measure of how incompatible observed data are with a specified statistical model, often a null hypothesis of no difference or no association. More precisely, it is the probability—assuming that model and its other assumptions are correct—of obtaining a test statistic at least as extreme as the one observed. It is not the probability that the null hypothesis is true, the probability that results happened “by chance”, or the size and importance of an effect. A small p-value can occur for a trivial effect in a huge sample, while an important effect can have a larger p-value in a small study. Good interpretation reports the estimate, confidence interval, design, assumptions and practical consequences alongside the p-value.
Key takeaways
- A p-value is calculated conditional on a statistical model.
- It does not give the probability that a hypothesis is true.
- The 0.05 threshold is a convention, not a law of nature.
- Sample size strongly affects how small a p-value can become.
- Effect size and uncertainty are essential for interpretation.
- Selective testing and reporting can create misleadingly small values.
Start with the hypothesis and test
A p-value has meaning only in relation to a test statistic, null hypothesis and assumptions. A two-sample test may assume independent observations and a specified distribution or use an approximation.
Before reading the number, identify what was compared, whether the test was one- or two-sided and which outcome was designated as primary.
The conditional probability
The calculation asks: if the null model were true, how often would data produce a statistic at least this extreme? It reasons from an assumed model to possible data.
It does not reverse that direction to calculate the probability of the model after observing data. Doing so confuses P(data or more extreme | model) with P(model | data).
What p = 0.03 means
Under the stated null model and assumptions, a result at least as extreme would occur about 3% of the time in repeated hypothetical samples. The exact definition of “extreme” comes from the test statistic and alternative hypothesis.
It does not mean there is a 3% probability the result is false, nor a 97% probability the research explanation is correct.
Why 0.05 is not a cliff
A result of 0.049 and one of 0.051 contain nearly the same evidence. Dividing them into discovery and failure exaggerates a tiny numerical difference. Thresholds can support pre-defined decisions but should be justified by consequences.
Reporting exact values and estimates avoids treating the threshold as a magical boundary. Disciplines with severe false-positive costs may require much stronger evidence.
Sample size and p-values
With a very large sample, a tiny departure from the null can yield a small p-value. With a small sample, even a substantial effect may be uncertain.
This relationship is explored in statistical power and sample size. The p-value cannot be read without knowing how much information produced it.
Effect size and practical meaning
A treatment difference, risk ratio or correlation describes magnitude. Practical significance asks whether that magnitude matters for a real decision.
As practical and statistical significance explains, a precisely estimated one-tenth-point change may be scientifically real yet operationally unimportant.
Confidence intervals
A confidence interval displays an estimate’s precision under a repeated-sampling procedure. It can show whether data remain compatible with meaningful benefit, negligible change or harm.
Reading the full range is more informative than checking only whether it contains zero. Confidence intervals also make wide uncertainty visible when a p-value is above a threshold.
A non-significant result
A p-value above 0.05 does not prove no effect. The data may be imprecise, the measurement noisy or the true effect small. Equivalence requires tests designed around a pre-specified meaningful margin.
The appropriate conclusion may be “the study did not distinguish among important benefit, no effect and harm” rather than “there was no difference”.
Multiple comparisons
If researchers test many outcomes, groups and models, some small p-values arise even when every null model is true. Adjustments, pre-registration and transparent reporting reduce this problem.
Exploratory analysis remains useful when labelled as exploration. Presenting only the most favourable comparison as if planned in advance misstates the evidence.
Optional stopping and researcher choices
Repeatedly checking data and stopping when p falls below 0.05 changes the long-run error rate unless the design accounts for sequential monitoring. Choices about exclusions and transformations can have similar effects.
An analysis plan written before seeing outcomes makes the decision process auditable. Sensitivity analyses show whether reasonable alternatives change the conclusion.
Study design and bias
A small p-value cannot repair confounding, poor randomisation, invalid measurement or missing data. It quantifies sampling variation within the chosen model, not every possible error.
A very precise online survey may still misrepresent the population because of sampling bias. Design quality comes before numerical significance.
Replication and prior evidence
One small p-value is not a permanent fact. Independent replication tests whether a result survives new participants, settings and researchers. Prior knowledge affects how surprising and credible a claim is.
A highly unexpected claim normally needs stronger and more varied evidence than a modest extension of an established mechanism.
Worked example
A trial of 30,000 people finds a 0.3-point average improvement with p = 0.004. The data are inconsistent with exact zero under the model, but the interval and decision threshold show the effect is too small to justify a costly rollout.
A pilot of 80 people estimates a five-point improvement with p = 0.08 and a wide interval. That study does not prove no benefit; it shows that the estimate is too uncertain for a confident decision.
A reporting checklist
State the hypothesis, test, sample size, estimate, units, interval and exact p-value. Explain planned and exploratory analyses, missing data and assumption checks.
Use plain language that separates observation from decision. “The estimated reduction was two cases per thousand” gives readers information that “highly significant” does not.
One-sided and two-sided tests
A two-sided test treats departures in either direction as relevant. A one-sided test focuses on a pre-specified direction and can produce a smaller p-value when the effect points that way. Choosing the direction after seeing data invalidates the intended error rate.
A one-sided design is defensible only when an opposite effect would genuinely be ignored or handled separately. In medicine, unexpected harm usually matters, which often supports two-sided analysis.
Model assumptions and diagnostics
Tests may assume independent observations, constant variance, random sampling or a particular error distribution. Clustered pupils within schools or repeated measurements from one patient violate simple independence. Appropriate multilevel or repeated-measures methods account for the structure.
Diagnostics and sensitivity analyses show whether conclusions depend on fragile assumptions. A precisely printed p-value from an unsuitable model is not more credible than a rounded value from a sound one.
Bayesian questions are different
Bayesian analysis combines a likelihood with a prior distribution to produce a posterior distribution for parameters. It can address questions about the probability of effect ranges, but answers depend on the prior and model.
A Bayesian result is not automatically objective or superior, just as a p-value is not automatically misleading. The method should match the question, and assumptions should be visible. Comparing approaches can show which conclusions are driven by data and which by modelling choices.
Teaching p-values with simulation
A classroom can repeatedly generate samples under a known null model and calculate a test statistic. The fraction at least as extreme as the observed statistic approximates the p-value and makes its conditional meaning concrete.
Changing sample size while keeping a tiny effect shows why statistical detectability is not practical importance. Simulation also demonstrates that 5% thresholds produce some false positives under repeated null experiments.
How to read a results table
Start with the outcome definition and units, then find the estimated difference and interval before looking at the p-value. Check the reference group, adjustment variables and sample size for that row. Missing observations can make rows rely on different participants.
If dozens of p-values appear, identify which analyses were planned and whether multiplicity was addressed. Footnotes often contain the assumptions needed to interpret the headline result, so they are part of the evidence rather than optional technical detail.
Conclusion
P-values can help assess how data compare with a statistical model, but they do not measure truth, importance or study quality. Interpretation improves when thresholds are pre-defined, exact values are reported and effect sizes and confidence intervals remain central. Design, measurement, multiplicity, bias and replication determine how much a result deserves to influence knowledge or action. The useful question is not simply whether p is below 0.05, but what effects are plausible and whether the evidence is strong enough for the claim being made.
Frequently asked questions
Does p = 0.05 mean a 5% chance the null is true?
No. It describes data under the assumed null model, not the probability of the hypothesis.
Is a smaller p-value always a larger effect?
No. Sample size and variability also affect the p-value.
What does p > 0.05 prove?
It does not prove no effect; it indicates that the data did not cross the chosen threshold under that test.
Should researchers report exact p-values?
Usually yes, alongside estimates, intervals and study details.
Can p-values be useful?
Yes, when correctly calculated and interpreted as one part of a broader evidence assessment.
