Practical significance and statistical significance answer different questions. Statistical significance asks whether an observed result would be unusual under a specified null model, often summarised with a p-value and a threshold. Practical significance asks whether the estimated effect is large enough to matter for health, education, policy, engineering, ecology or another real decision. With a huge sample, a tiny and unimportant difference can be statistically significant. With a small sample, an important effect can remain statistically uncertain. Good analysis therefore reports the effect size, its confidence interval, measurement quality, costs, benefits and relevant decision threshold. A p-value is one piece of evidence, not a label that turns a result into important truth.
Key takeaways
- Statistical significance does not measure importance.
- Practical significance depends on effect size and decision context.
- Large samples can make very small effects statistically detectable.
- Confidence intervals show which effect sizes remain plausible.
- A non-significant result is not proof of no effect.
- Thresholds should be chosen before seeing results where possible.
What statistical significance means
A null hypothesis commonly states that a population effect equals zero or another reference value. A p-value is the probability, assuming the null model and other assumptions are correct, of obtaining data at least as incompatible with that model as the observed data.
It is not the probability that the null hypothesis is true, and one minus the p-value is not the probability that the research hypothesis is correct. Calling p below 0.05 “significant” is a convention, not a natural law.
What practical significance means
Practical significance concerns magnitude and consequences. A school may ask whether a programme improves learning enough to justify teacher time. A river manager may ask whether a pollutant reduction changes ecological status. A doctor may compare a treatment benefit with harms and cost.
The same numerical effect can matter differently in different settings. A one-millimetre tolerance could be trivial for landscaping and critical for an aircraft component. Practical thresholds should be justified by subject knowledge.
The role of effect size
An effect size expresses how large a difference or relationship is. It may be a difference in means, risk ratio, correlation, percentage-point change or standardised measure. Raw units are often easiest for decision-makers to interpret.
Standardised effects help compare different scales but can hide the real-world unit. Reporting both may be useful. An average effect should also be separated from variation among people, places or time periods.
Why sample size changes significance
Standard errors generally shrink as sample size grows. A very large study can estimate a tiny effect precisely and produce a small p-value. That result may be real yet practically negligible.
A small study may estimate a large effect with wide uncertainty and a p-value above the chosen threshold. As statistical power and sample size explains, absence of statistical significance may reflect imprecision rather than absence of a meaningful effect.
Confidence intervals connect the questions
A confidence interval presents a range generated by a repeated-sampling procedure and shows the precision of an estimate. Rather than asking only whether zero lies inside, compare the interval with thresholds for meaningful benefit and harm.
If every plausible value is too small to matter, the effect is practically unimportant even if statistically distinguishable from zero. If the interval includes large benefit and harm, the study is inconclusive. Understanding confidence intervals encourages this range-based reading.
Measurement quality matters
A precise estimate of a biased or poorly defined measure is not useful. Instrument calibration, outcome definition, missing data and observer decisions can shift results. Statistical calculations do not automatically absorb systematic error.
Compare the effect with measurement uncertainty and real change. A small difference below the instrument’s reliable resolution may not support an operational decision despite an impressive sample size.
Statistical significance in experiments
Random allocation helps separate a treatment effect from confounding, but attrition, non-compliance and selective reporting still matter. Researchers should define the primary outcome and analysis plan before looking at results.
Multiple testing increases the chance that at least one comparison crosses a threshold. Corrections, replication and transparent reporting are more informative than selecting only statistically significant outcomes.
Observational data and causation
A statistically significant association in observational data can arise from confounding, selection or reverse causation. Increasing sample size reduces random uncertainty but does not remove those biases.
Representative sampling also matters. A precise estimate from the wrong population may not generalise, a problem examined in sampling bias. Practical decisions need evidence relevant to the people or places affected.
Worked example: a classroom programme
Suppose 20,000 pupils use a digital exercise and score 0.4 percentage points higher on a test, with p below 0.001. The result is statistically significant because the estimate is precise. The school must still ask whether 0.4 points changes learning, progression or equity enough to justify licensing and training.
A confidence interval of 0.2 to 0.6 points rules out a large average improvement. Subgroup effects and implementation costs are examined, but exploratory subgroup findings are labelled as such. The evidence may support “small detectable effect” rather than “programme transforms attainment”.
Decision thresholds and trade-offs
A minimum important difference should be specified from domain evidence, stakeholder needs and consequences. Decisions may also use cost-effectiveness, risk tolerance or legal limits. These are value-laden choices informed by statistics, not quantities derived from a p-value alone.
Uncertainty can justify further research, cautious adoption or monitoring rather than a simple yes/no answer. The cost of a false positive may differ greatly from the cost of missing a real benefit.
Better reporting
Report sample size, estimate, units, confidence interval, study design and assumptions. Give exact p-values where appropriate rather than only stars or threshold labels. Explain missing data and all planned outcomes.
Use plain language: “the estimated reduction was two cases per 1,000, with a plausible range from one fewer to four fewer.” That sentence lets readers judge size and uncertainty more directly than “highly significant”.
Equivalence and non-inferiority questions
Sometimes the aim is not to prove a difference but to show that any difference is smaller than a pre-specified practical margin. Equivalence testing asks whether the plausible effect lies inside both positive and negative limits. A non-inferiority trial asks whether a new option is not unacceptably worse than a comparator, perhaps because it is cheaper or safer.
Failing to reject a usual zero-difference test does not establish equivalence. A wide confidence interval may contain both important benefit and important harm. The practical margin must be justified before examining results, because moving it afterwards can make a disappointing study appear successful.
Heterogeneous effects
An average improvement can conceal different outcomes among groups. A programme might help beginners while offering little to advanced students, or reduce average pollution while increasing exposure in one neighbourhood. Subgroup analysis is useful when supported by theory and enough data, but searching many divisions creates unstable findings.
Practical significance therefore includes distribution as well as the mean. Decision-makers ask who benefits, who bears costs and whether the implementation changes inequalities. Confidence intervals for subgroup differences are more informative than declaring one group significant and another not significant.
Replication and cumulative evidence
One study’s threshold crossing should not dominate a field. Replication, systematic review and meta-analysis combine evidence while examining design quality and heterogeneity. Several small, compatible effects may be practically important at population scale; several biased studies do not become reliable merely because their p-values are pooled.
Good cumulative reasoning keeps outcomes in real units, checks publication bias and distinguishes confirmatory from exploratory work. Statistical significance then becomes one diagnostic within a wider account of magnitude, uncertainty, mechanism and feasibility.
A checklist for judging a reported effect
Write the result in its original units, not only a p-value or percentage change. Record the estimate, confidence interval, sample size and comparison group. Identify the minimum effect that would change a real decision and check whether the interval lies above, below or across it. A relative reduction should be paired with absolute risk so readers can see the baseline.
Then examine design, missing data, measurement validity and multiplicity. Ask whether the population matches the decision and whether costs or harms were measured. A precise small average may be useful at national scale, while a larger uncertain effect may justify another study. The correct judgement combines numerical evidence with explicit consequences rather than treating “significant” as a synonym for important.
Conclusion
Statistical significance describes how data compare with a model; practical significance describes whether the size and consequences of an effect matter. Neither replaces sound design, measurement or judgement. Effect sizes and confidence intervals connect numerical evidence to decision thresholds, while costs, harms and population relevance supply the context. The most useful question is rarely “Was p below 0.05?” It is “What effects remain plausible, and would those effects change what we should do?”
Frequently asked questions
Can a tiny effect be statistically significant?
Yes. Very large samples can detect effects too small to matter in practice.
Can an important effect be non-significant?
Yes. A small or noisy study may have a wide interval that includes meaningful effects.
Is p = 0.04 much stronger than p = 0.06?
Not usually. The values are close and should not be split into certain success and failure without considering the full evidence.
How is practical significance chosen?
By subject knowledge, stakeholder priorities, costs, harms and a justified minimum important difference.
Should p-values be abandoned?
They can be useful when correctly interpreted, but should be reported with estimates, intervals and study context.
