Sampling bias occurs when the method used to select or retain observations makes some members of the target population systematically more likely to appear than others, and that imbalance is related to the quantity being estimated. A survey of thousands of volunteers can therefore be more misleading than a smaller probability sample. The problem is not simply that a sample differs from a population by chance; random sampling error is expected and can be estimated. Bias is directional distortion built into coverage, recruitment, response or measurement. Online polls, street interviews, wildlife surveys and medical studies can all suffer from it. Researchers reduce risk by defining the population, building an appropriate sampling frame, selecting units with known probabilities, following up non-respondents and using justified weights. No method guarantees perfection, so good reports explain who was eligible, who was reached, who responded and how uncertainty was assessed.
Key takeaways
- Sampling bias is systematic under-representation or over-representation, not ordinary random variation.
- A very large biased sample can give a precise-looking but wrong estimate.
- Coverage, self-selection, non-response and exclusion can each distort results.
- Probability sampling gives units known selection chances and supports inference.
- Weighting can reduce known imbalance but cannot automatically repair unknown differences.
- The target population, frame, response rate and question wording all need scrutiny.
Population, sample and sampling frame
The target population is the full group about which a study seeks conclusions: all eligible voters, households in a country, trees in a forest or water samples in a catchment. A sample is the subset actually studied. The sampling frame is the operational list or map from which that subset is selected.
A perfect frame covers every target unit once, but real frames can omit recent movers, duplicate addresses, exclude people without internet access or define geographic units imperfectly. Frame quality matters before any sophisticated analysis begins. A random draw from an incomplete list is random only within that list.
Researchers must also define the unit. Sampling households and interviewing one adult is different from sampling adults directly. If selection within a household favours the person who answers the door, working patterns and age may affect inclusion.
Bias is different from sampling error
Two well-designed random samples will contain different people and produce slightly different estimates. That is sampling variability. Standard errors and confidence intervals describe the uncertainty generated by the sampling design under stated assumptions. Larger probability samples usually reduce this random error.
Bias does not necessarily shrink when more observations are collected by the same flawed route. If a voluntary poll attracts mostly people with strong views, adding thousands more volunteers can make the estimate very stable for volunteers while remaining unrepresentative of everyone else.
Earth Site’s guide to measurement uncertainty and real change helps separate noisy estimates from systematic error. A narrow margin of error normally covers sampling variation, not every possible coverage, non-response or wording problem.
Common forms of sampling bias
Coverage bias
Coverage bias appears when the frame misses or duplicates parts of the population. A telephone survey that excludes people without the selected type of service may differ systematically by age, income or location. A wildlife count along roads may over-sample edge habitats.
Self-selection bias
In an open online poll, people choose whether to participate. Strong interest, anger, free time or familiarity with the website can affect participation. The poll may describe its respondents accurately while saying little about the public.
Non-response bias
A probability sample can become biased if selected units do not respond and their answers would differ from respondents. Response rate is useful but not a complete diagnosis: a lower rate can sometimes be adjusted well, while a higher rate can still hide systematic gaps.
Survivorship and convenience
Studying only surviving companies misses failures. Sampling the nearest stream sites may miss polluted or inaccessible reaches. Convenience reduces cost but weakens the basis for generalisation.
Random, systematic and stratified sampling
In simple random sampling, each unit on the frame has an equal known chance of selection. The method prevents a researcher from hand-picking favourable cases, though it does not repair a poor frame or non-response.
Systematic sampling selects every kth unit after a random start. It is efficient, but hidden periodic ordering in the list can cause trouble. For field transects, regular spacing may capture gradients well, yet it should be tested against patterned landscapes.
Stratified sampling divides the population into groups such as region, age band or habitat, then samples within each. This ensures representation and can improve precision. If small groups are deliberately over-sampled, weights restore their correct population contribution. Cluster sampling selects groups—schools or neighbourhoods—then studies units within them, lowering travel cost but often increasing similarity among observations.
What weighting can and cannot do
A survey weight tells how much population representation an observation carries. Design weights reflect selection probabilities. Non-response adjustments and calibration may align respondents with known population totals for age, sex, region or other variables. The Office for National Statistics and other agencies publish methods so users can understand these steps.
Weighting works best when adjustment variables are related both to response and to the outcome. It cannot guarantee that respondents and non-respondents match on unmeasured characteristics. Extreme weights can make estimates depend heavily on a few responses and increase variance. Analysts may trim weights, but trimming trades variance against possible bias.
Published results should state whether figures are weighted, which controls were used and how complex design affects uncertainty. Treating a weighted sample as if it were a simple random sample can understate error.
Sampling in geography and environmental science
A population is not always people. Soil studies choose plots and depths; river surveys choose sites and dates; satellite validation chooses ground locations. Spatial autocorrelation means nearby observations are often more similar than distant ones, so many closely packed samples may supply less independent information than their count suggests.
Accessibility bias is common. Flat, safe and road-adjacent locations are easier to measure than steep, remote or private land. If environment differs with accessibility, estimates shift. Researchers use random spatial points, stratification by land cover and documented replacement rules.
Population geography adds another distinction: population density and population distribution are not interchangeable. Sampling equal land areas over-represents sparsely inhabited places if the question concerns people; sampling people may be correct for opinion but wrong for land exposure.
Sampling bias, correlation and causation
A biased sample can create, hide or reverse an association. An exercise app may show that users are unusually active because active people choose the app, not because downloading caused the activity. Hospital patients differ from all people with a condition because admission depends on severity and access.
This connects to correlation versus causation. Representativeness and causal identification are separate. A representative opinion survey can estimate prevalence without proving causes. A randomised trial may estimate a treatment effect in volunteers but still require care when generalising to another population.
The sampling design should match the intended claim. Describing one class, village or mine does not require pretending it is statistically representative if it is presented as a case study. Problems arise when conclusions silently expand beyond observed cases.
How to evaluate a survey or poll
- Target: Who exactly is the result supposed to represent?
- Frame: How could eligible units enter the sample, and who was missing?
- Selection: Were probabilities known, or could anyone volunteer?
- Response: How many selected units participated, and did response differ among groups?
- Measurement: Were questions neutral, understandable and asked consistently?
- Adjustment: What weights and uncertainty calculations were used?
- Timing: When was fieldwork done, and could events during it change answers?
A polished chart cannot substitute for these answers. Ask for the method and questionnaire, not only the headline percentage. Scientific knowledge grows through transparent and repeatable evidence, as discussed in hypotheses, theories and scientific evidence.
A numerical example: large does not mean representative
Suppose a town has 8,000 adults under 50 and 2,000 aged 50 or over. In reality, 40 per cent of the younger group and 70 per cent of the older group support a proposal. The population result is 46 per cent: 3,200 plus 1,400 supporters out of 10,000 adults.
An online poll attracts 2,000 responses, but older adults are unusually engaged and provide half of them. The poll reports 55 per cent support: 400 supporters among 1,000 younger respondents and 700 among 1,000 older respondents. Its random-looking margin of error may be small, yet the estimate is nine percentage points too high because the age mix is wrong.
Weighting the age groups back to 80 and 20 per cent would recover 46 per cent if respondents within each age group are representative. If politically enthusiastic people were also more likely to respond within both groups, age weighting alone would not fix the problem. The example shows both the value and the limit of calibration.
Conclusion
Sampling bias is a problem of connection: the route from population to observed cases favours some units in a way related to the answer. Bigger datasets do not automatically solve it. Strong studies define their target, use an appropriate frame, control selection, investigate non-response and publish adjustments with uncertainty. Readers can make better judgements by asking who had a chance to be counted and who actually was. Those questions apply as much to a river survey or wildlife transect as to an election poll.
Frequently asked questions
Is a random sample always representative?
Not exactly in every realised sample. Random samples vary by chance, but probability selection supports unbiased estimation over repeated samples and allows sampling uncertainty to be calculated. Frame gaps and non-response can still bias results.
How large should a sample be?
It depends on desired precision, population variability, design, subgroup estimates and expected response. Size cannot compensate for a selection method that systematically excludes relevant groups.
Does a low response rate prove bias?
No, but it raises concern. Bias depends on whether response is related to the outcome after adjustment. Researchers should analyse respondent differences and follow-up evidence.
Can weighting fix an online poll?
Usually not fully if selection probabilities are unknown and volunteers differ on unmeasured factors. Weighting known demographics may improve balance without making it a probability sample.
What is the difference between sampling bias and question bias?
Sampling bias concerns who is observed. Question or measurement bias concerns how answers are elicited or recorded. A study can suffer from either or both.
