Statistical power is the probability that a planned statistical test will detect an effect of a specified size when that effect really exists under the model being used. An underpowered study has a high chance of a false negative: failing to reject a null hypothesis even though an important difference is present. Power usually increases with sample size, a larger true effect, lower variability and more efficient study design. It is not a general score for scientific quality. Thousands of biased observations can produce a precise answer to the wrong question, while a small but well-designed study may still provide useful estimates with appropriately wide uncertainty. Sample-size planning therefore begins with the decision that matters, the minimum effect worth detecting, acceptable error risks and realistic variability—not with a universal rule that a particular number of participants is enough.
Key takeaways
- Power concerns detecting a specified effect when it exists.
- Larger samples usually increase power and precision.
- Small effects require more information to distinguish from noise.
- Significance level and power represent different error risks.
- Biased sampling is not cured by a large sample.
- Planning should account for clustering, drop-out and multiple outcomes.
The four ingredients of a power calculation
A conventional calculation links effect size, sample size, variability and significance level. Specify three and assumptions about the test, and the fourth can be calculated. The effect should be expressed on the outcome’s real scale before conversion to a standardised measure.
“Effect size” here means the difference the study aims to detect, not the observed result after data collection. A tiny effect may be real but unimportant. Planning around the smallest practically important difference aligns statistics with the research question.
False positives and false negatives
The significance level, often called alpha, limits the long-run probability of rejecting a true null hypothesis under the test assumptions. Power equals one minus beta, where beta is the probability of missing the specified effect. Lowering alpha while everything else stays fixed usually reduces power.
These rates do not state the probability that a particular published conclusion is true. That also depends on study design, model assumptions, prior plausibility, selective reporting and the evidence pattern beyond one threshold.
Why larger samples help
Sampling variation makes small studies’ estimates fluctuate widely. More independent observations usually narrow the sampling distribution and confidence intervals, helping separate a true signal from random variation. Gains show diminishing returns: halving a standard error commonly requires roughly four times as many independent observations.
Independence is crucial. Ten readings from each of ten people do not necessarily provide the same information as one reading from one hundred independent people. Repeated measurements can be valuable, but the analysis must model their correlation.
Effect size and variability
A five-point improvement is easier to detect when scores normally vary by two points than when they vary by forty. Better measurement can increase power by reducing noise. Pairing before-and-after observations can help when within-person change is more stable than comparisons between unrelated people.
Standardised effect sizes express a difference relative to variability, but they can hide practical units. Reports should return to kilograms, test points, degrees or disease risk so readers can judge importance.
One-sided tests, two-sided tests and multiple outcomes
A one-sided test assigns the rejection region to one direction and may provide more power there, but it cannot be chosen after seeing the result. Harm in the unexpected direction may still matter. Two-sided tests are often appropriate when either direction is scientifically relevant.
Testing many outcomes, subgroups or model versions increases opportunities for apparently significant findings. Adjustments, pre-specified primary outcomes and replication address this problem. Simply increasing sample size without controlling analytical flexibility can make trivial or selected patterns look convincing.
Power is not protection from bias
If a survey systematically excludes people with limited internet access, adding more online responses tightens precision around the online population rather than representing everyone. This is the distinction explained in sampling bias.
Confounding and measurement error create similar problems. A large observational study can show a precise association without establishing cause, as discussed in correlation and causation. Design quality comes before calculation.
Clustered and complex designs
Pupils within a school, patients within a hospital and measurements within a household tend to resemble one another. Clustering reduces the effective independent information compared with the same total number sampled independently. Power calculations use an intracluster correlation and expected cluster size to allow for this.
Unequal allocation, stratification, covariates, repeated measures and survival outcomes require methods matched to their analysis. A generic online calculator can mislead if its assumptions do not match the eventual model.
Drop-out and missing data
Researchers often recruit extra participants to allow for expected attrition. If 20 per cent may leave, simply dividing the required complete sample by 0.8 gives a basic recruitment target. But missingness related to treatment or outcome can bias results, not merely reduce power.
Retention planning, accessible procedures and reasons for withdrawal deserve attention. Inflating a target cannot justify burdensome recruitment or hide unrealistic assumptions.
Worked example: planning a classroom trial
A team wants to detect a five-point improvement in a standardised assessment. They set alpha at 0.05, power at 0.80 and estimate variability from comparable pupils. The initial calculation assumes individual randomisation, but teaching is delivered by class. Pupils in the same class share teachers and context, so the team revises the design for clustering and recruits more classes.
They also specify one primary outcome, allow for absence and decide that a smaller improvement would not justify the cost. The final protocol follows principles of a fair comparison. If the result is inconclusive, the interval and achieved data—not a retrospective declaration that there was “no effect”—show what remains plausible.
After the study: avoid observed power
Calculating “observed power” from the effect seen in the same data usually adds little: it is largely a transformation of the p-value and treats an uncertain estimate as true. Report the estimate, interval, sample flow and deviations from the plan.
If recruitment fell short, explain it. Sensitivity analyses can show what effect sizes the completed study could estimate with useful precision. Honest uncertainty is more useful than retrofitting a target to make the study appear adequate.
Precision-based planning and estimation
Not every study is built around a hypothesis test. Researchers may choose sample size so a confidence interval is expected to have a useful width. A survey estimating a proportion within a few percentage points asks a different question from a trial designed to detect a treatment difference.
Precision planning requires an expected variability or proportion and often a conservative assumption. Finite-population corrections can matter when sampling a substantial share of a small population. Complex survey weights and clustering can widen uncertainty relative to a simple random sample.
Focusing on estimation encourages researchers to state what range would support a decision. If every plausible value after the study leads to the same action, extreme precision may be unnecessary. If the interval straddles a safety threshold, additional data may have high value.
Ethics, feasibility and simulation
Recruiting too few participants can expose people to burden without a realistic chance of answering the question. Recruiting far more than necessary can also waste resources or expose extra participants to risk. Ethical review therefore considers the justification for sample size alongside consent and safety.
When formulas are unsuitable, simulation can generate data under planned assumptions, run the intended analysis and count how often it reaches a criterion. This handles missingness, unequal clusters and complex models, but only if scenarios reflect reality. Code and assumptions should be documented.
Pilot data can inform variability and feasibility, yet small pilots estimate standard deviations imprecisely. External studies and conservative ranges may be better. Sensitivity calculations showing several plausible assumptions are more honest than presenting one exact sample target as though it were measured rather than planned.
Wider significance and responsible use
Low power can distort a literature because only unusually large estimates cross a publication threshold. Published effects from small studies may therefore exaggerate the typical effect even when each analysis followed its stated test. Registration, full reporting and synthesis of inconclusive studies reduce this selection.
High power can make negligible differences statistically significant. A sample of millions may estimate a change of a few hundredths with great precision, but the practical meaning still depends on outcomes, costs and alternatives. Statistical detection and importance are separate judgements.
Power calculations are conditional forecasts, not guarantees. Recruitment, adherence, variability and model fit can differ from assumptions. Monitoring these design quantities without inspecting comparative outcomes can protect study integrity, while transparent final reporting explains the gap between planned and analysed samples.
Further questions for investigation
Sequential and adaptive designs can examine accumulating information without using ordinary repeated significance tests. Pre-specified stopping boundaries protect error rates and may allow a trial to stop for benefit, harm or futility. Flexibility comes from design, not from checking until a preferred result appears.
Replication planning should not merely copy an original small sample. A replication can target a range of plausible effects, including publication-bias-adjusted estimates, and use higher power. Differences in population or procedure should be planned as tests of generality rather than explained only after results differ.
Conclusion
Power and sample size connect scientific goals to the amount and structure of information a study needs. They require assumptions about effect, variability, errors and analysis, all of which should be justified before results are known. More data can improve precision, but it cannot rescue biased selection, confounding or poor measurement. A strong plan combines calculation with design, feasibility, ethics and a clear definition of what difference would matter.
Frequently asked questions
Is 80 per cent power always required?
It is a common convention, not a universal law. The appropriate target depends on consequences, feasibility and the evidence needed.
Does a non-significant result prove there is no effect?
No. Examine the estimate and confidence interval to see which effects remain compatible with the data.
Can a sample be too large?
A very large study can detect negligible differences and waste resources if the question and practical threshold are unclear.
Why does clustering require more participants?
People within clusters share conditions, so their observations contain less independent information than the same number sampled separately.
Should sample size be changed during a study?
Only under a pre-specified or statistically valid adaptive plan that protects error rates and prevents knowledge of results from distorting decisions.
