Survey (human research)

• 1. Introduction

  ›

  Survey research traces its formal origins to a committee in 1838, when the Statistical Society of London printed a list of questions designed to document the complete and impartial history of strikes. That moment planted the seed for what became one of the most widely used tools in modern science. Today, surveys shape healthcare policy, determine which television programs get renewed, and inform how political campaigns spend their money. Yet a good survey is far harder to build than it looks. What makes a question biased? How do researchers turn answers from a small sample into claims about millions of people? And why might a respondent's stated behavior diverge sharply from what they actually do? Those are the questions this documentary will explore.

• 2. What a Survey Actually Is

  ›

  A survey is, at its core, a predetermined set of questions given to a sample of people. The defining goal is to extract specific data from a particular group, then generalize those findings to a larger population. Psychologists and sociologists use them to analyze behavior. Media organizations use them to evaluate political candidates and guide advertising strategy. Healthcare researchers use them to measure gaps between recommended clinical guidelines and what practitioners actually do in practice.

  The logic depends heavily on sample quality. A representative sample, one that mirrors the broader population of interest, lets researchers describe attitudes far beyond the individuals who answered. Without that representativeness, the entire enterprise breaks down. A survey can cover very narrow topics or pursue broad, widespread goals, but the underlying discipline of sampling applies in either case.

  Bias in question wording is a persistent hazard. Suggestive words, or language that different groups interpret differently, can skew results. When such language cannot be avoided, researchers must apply special analysis methods to prevent inaccurate conclusions from slipping through.

• 3. A History of Counting People

  ›

  The United States census has been held every ten years since 1790, making it the most famous public survey in the country. Its purpose is to count all persons and gather demographic data on age, ethnicity, and household relationships. The United Nations recommends that all nations conduct population and housing censuses at least every decade, defining essential features as individual enumeration, universality within a defined territory, simultaneity, and defined periodicity.

  The application of probability sampling in the 1930s was a turning point. It transformed surveys from ad hoc information-gathering into a standard tool for empirical research across the social sciences, marketing, and official statistics. That methodological shift gave researchers a principled way to draw defensible conclusions from a finite group of respondents.

  Nielsen ratings, carried out since 1947, show how the logic of public surveying extended into commercial life. Nielsen tracks radio, television, internet, and print viewing habits, and those results feed directly into commissioning decisions for broadcasters. Some Nielsen data is localized to give marketing firms more granular targeting information.

• 4. The Spread of Survey Modes

  ›

  Telephone surveys arrived at least as early as the 1940s, offering interviewers a way to reach large national or international sampling frames at reasonable cost. Interviewer presence tends to raise response rates, but it also introduces bias: research has found that some respondents discuss sensitive topics more openly with an interviewer of a particular gender.

  Mail surveys extended the reach further, allowing questionnaires to travel to households without any interviewer involvement. That absence of an interviewer removes one source of bias and lets respondents answer at their own convenience, including breaking up long surveys or checking records before replying. The trade-off is time: the field period for a mail survey often runs several months before returns are complete and analysis can begin.

  The development of the internet in the late twentieth century opened a new channel entirely. By 2006, online survey research accounted for 20% of global data-collection expenditure according to ESOMAR. Web surveys can collect and process data in little more than a month, a fraction of the time required by mail. The shift from traditional paper-and-pencil interviewing to computer-assisted interviewing also changed how questions are delivered: face-to-face surveys using computers (CAPI), telephone surveys (CATI), and mail surveys (CASI, CSAQ) all have distinct protocols.

  Over 50% of surveys today are opened on mobile devices. A World Bank study in Guatemala found a 71% decrease in cost when mobile data collection replaced a paper-based approach. Mobile surveys also reach populations that were previously hard to contact, and SMS surveys can deliver 80% of responses in less than two hours.

• 5. Online Surveys: Promise and Problems

  ›

  Online surveys offer capabilities that no paper questionnaire can match: complex skip patterns invisible to respondents, pop-up help text, multimedia elements, and the ability to preload known information about a respondent. Employees in multiple studies have indicated a preference for online survey formats over paper-and-pencil versions. Studies comparing probability-based web surveys to face-to-face surveys with show-cards found similar levels of measurement quality; telephone surveys performed worse on that measure.

  Response rates are the Achilles heel of the online mode. They vary enormously, from less than 1% in enterprise surveys sent by email to nearly 100% in specific membership surveys. Researchers can try to lift rates by offering monetary incentives, making multiple follow-up contacts, and keeping questionnaires as simple as possible. But incentives carry a risk of their own: respondents who are paid to answer may not produce unbiased data. Publicizing what is actually done with survey results has been identified as one of the most concrete ways to motivate honest, sustained participation.

  Acquiescence bias is a subtler problem. Many people have acquiescent personalities and tend to agree with statements regardless of content, partly because they perceive the question-asker as an expert. Sampling also creates structural difficulties online: not all members of the general population have internet access, and there are no comprehensive email directories that could serve as a sampling frame. Both issues can significantly distort data quality if they go unreported.

• 6. When Respondents Do Not Tell the Truth

  ›

  Survey researchers generally accept answers as truthful, but direct observation has repeatedly shown that the link between self-reported attitudes and actual behavior, while positive, is not always strong. Dishonesty is particularly pronounced in sex-related questions. Men frequently amplify their reported number of sexual partners, while women tend to understate theirs. Researchers must apply caution before treating self-reports as reliable proxies for real-world conduct.

  Measurement error adds another layer of distortion. Correlation coefficients between two variables in survey data can be skewed by error in how those variables were measured, leading to biased substantive conclusions. Researchers are expected to correct coefficients for measurement error before drawing inferences.

  Correlation itself is often misread. When two survey variables move together, that association does not establish that one causes the other. Path analysis, a statistical technique used with correlational data, helps researchers identify mediator variables that explain a correlation and moderator variables that affect its direction or strength. A spurious relationship, where an apparent link between two variables is actually driven by a third, is a persistent hazard in interpreting survey results.

• 7. Large-Scale Surveys in Practice

  ›

  Some surveys are designed to run for decades and track the same people over time. The Panel Study of Income Dynamics in the United States started with over 18,000 nationally representative individuals and has collected data from the same families and their descendants since 1968, with more than 9,000 individuals still involved as of 2009. The Socio-Economic Panel in Germany began in 1984, re-interviews adult household members every year, and by 2007 covered about 12,000 households and more than 20,000 adult persons.

  The Integrated Household Survey in the United Kingdom brings together multiple existing surveys under one umbrella. With about 340,000 respondents, it stands as the largest collection of social data in the UK after the national census. The National Survey of Family Growth, conducted by the National Center for Health Statistics within the Centers for Disease Control and Prevention, gathered 22,682 interviews during its 2006-2010 cycle, focusing on fertility, family structure, and demographic trends in the United States.

  The Generations and Gender Survey reaches across continents: it has collected at least one wave of data in 19 countries, spanning several European nations as well as Australia and Japan, with an average of 9,000 respondents per country. Understanding Society, a longitudinal study of UK households formerly known simply as UK households, draws on a sample of 40,000 households, roughly 100,000 individuals.