Survey methodology

• 1. Introduction

  ›

  A survey of a population that is 75 percent female and 25 percent male might still return a sample that comes out 40 percent female and 60 percent male. Every number that survey produces is now built on a distortion, since women are underrepresented and men are overrepresented, a problem survey methodologists call selection bias. Survey methodology is defined simply as 'the study of survey methods', but underneath that plain definition sits a discipline built to answer a much harder question: how do you ask a limited group of people questions and still trust that their answers describe a much larger population? The field spans political polls, public-health surveys, market research, government surveys and censuses, anywhere someone needs a statistical inference about people most of whom were never actually asked anything; even a census, which surveys an entire population rather than a sample, still relies on the same questionnaires, interviewers and follow-up techniques as a smaller survey. What turns a badly built questionnaire into a reliable one, and why might a fax machine reach some respondents better than the mail ever could?

• 2. Building a Representative Sample

  ›

  A survey draws its sample from a sampling frame, typically a list of population units, though in area sampling the frame can be a map divided into geographic units instead. Each member of that population is called an element, and the survey's real goal is never to describe the sample itself but to generalize accurately to the larger population it was drawn from, whether that population is an entire country, a narrower group within it, the membership list of a professional organization, or the students enrolled in a single school system. The people who answer are called respondents, and depending on the question, their answers may describe themselves as individuals or speak for a household, an employer, or another organization they represent.

  Achieving that generalization means avoiding selection bias, the problem that arises whenever the selection procedure over-represents or under-represents some meaningful slice of the population. Researchers commonly guard against it with stratified random sampling, dividing the population into sub-groups called strata and then drawing random samples from each stratum, or otherwise selecting elements on a proportional basis. Survey methodologists also treat errors as a cost problem: cutting errors can mean improving quality within a fixed budget, or cutting costs while holding quality steady, and the field itself is both a scientific discipline, studying where errors come from, and a profession, made up of practitioners who design surveys to keep those errors down.

• 3. Modes of Collecting Data

  ›

  Telephone calls, mailed questionnaires, online surveys, mobile surveys, personal in-home interviews and mall or street intercept surveys are the main ways researchers administer a survey, sometimes combined into mixed-mode designs. The choice between them depends on cost, how well a mode covers the target population, how flexible it allows questions to be, how willing respondents are to participate, and how accurate the resulting answers turn out to be. Different modes also produce different mode effects, meaning the format itself can change how a respondent answers the same question.

  Before any mode is chosen, a survey methodologist faces a set list of decisions: how to identify and select potential sample members, how to reach people who are hard to contact or reluctant to answer, how to evaluate and test the questions themselves, which mode to use for posing them, how to train and supervise any interviewers involved, how to check the resulting data files for accuracy and internal consistency, how to adjust the final estimates for errors that turn up, and whether to complement the survey with other data sources.

• 4. Three Study Designs

  ›

  A cross-sectional study draws a sample once and describes the characteristics of a population at that single moment, which makes it useful for description but not for identifying the causes behind those characteristics, since its design is only predictive and correlational. A successive independent samples design instead draws multiple random samples from the same population at different times, which can reveal how a population changes overall, though not how any individual within it changed, since the same people are not surveyed twice; for the comparison to hold, each sample must be equally representative and the questions must be asked identically each time.

  A longitudinal study goes further, surveying the very same random sample repeatedly over time, which lets researchers assess why individual responses shifted rather than just that they did, making it well suited to studying naturally occurring events, such as divorce, that cannot be tested experimentally. Longitudinal studies are also expensive and difficult to sustain, since it is harder to find people willing to commit to a study lasting months or years than to a 15-minute interview, and participants routinely drop out before the final assessment. When anonymity is required, researchers sometimes rely on a self-generated identification code built from details like a respondent's month of birth and the first letter of their mother's middle name, or newer approaches built around a question like the name of a person's first pet. Because attrition is rarely random, researchers can compare those who left a study against those who stayed to check whether the two groups differ in ways that would skew the results.

• 5. Writing Questions That Work

  ›

  A survey is worthless if its questionnaire is written badly, no matter how carefully the sample behind it was drawn. Questionnaires are also the tool researchers use to measure demographic variables, such as ethnicity, socioeconomic status, race and age, and to run self-report scales, among the most widely used instruments in psychology, that capture people's preferences, attitudes and judgements on a given topic. Six steps go into building a questionnaire likely to produce reliable, valid results: deciding what information to collect, deciding how to administer it, drafting it, revising it, pretesting it, and finally editing both the questionnaire and the procedures for using it. Reliability and validity are not the same thing. A reliable measure produces consistent results across repeated tests, which researchers check with test-retest reliability, comparing where the same people rank in the score distribution on two separate occasions rather than expecting identical scores; reliability tends to improve when a scale uses many items, when the trait being measured varies widely across the sample, and when instructions are clear and distractions are limited. A valid measure, by contrast, actually measures the theoretical concept, or construct, it was built to measure.

  Free response questions, which are open-ended, give respondents more flexibility but are harder to record and score, requiring extensive coding, while closed, mostly multiple-choice questions are easier to code but limit spontaneity. Survey researchers are advised to keep question wording simple and direct, generally under twenty words, to avoid leading or loaded phrasing, and to word some items in a construct in the opposite direction to guard against response bias. Question order matters too: self-administered questionnaires should put the most interesting questions first and demographic questions near the end, while telephone or in-person interviews should open with demographic questions to build the respondent's confidence, since an earlier question can prime how people answer a later one.

  Translating a questionnaire is not a mechanical word-for-word process. The TRAPD model, standing for Translation, Review, Adjudication, Pretest and Documentation, was originally developed for the European Social Surveys and is now widely used across the global survey research community, even when it isn't labeled as such. Sociolinguistics supplies a theoretical framework that complements TRAPD, holding that a translated questionnaire only achieves the same communicative effect as the original if the translation reflects the social practices and cultural norms of the target language, not just its vocabulary.

• 6. Getting People to Respond

  ›

  An advance letter, sent before a phone call or in-person visit, announces the coming survey, briefly describes its topic, and thanks the respondent in advance for their cooperation, a technique recommended for reducing nonresponse. Interviewers who are thoroughly trained in asking questions, using computers and scheduling callbacks tend to see better results, as does opening every interview with a short introduction naming the interviewer, their institute, and the length and goal of the survey; making clear that nothing is being sold has also been shown to modestly raise response rates. A questionnaire itself helps its own cause when its questions are clear, non-offensive and easy for the subjects being studied to answer.

  A 1996 literature review found only mixed evidence that shorter surveys actually improve response rates, concluding other factors often matter more. A larger 2010 study of 100,000 online surveys found response rates dropped by about 3 percent once a survey reached 10 questions and by about 6 percent at 20 questions, though the rate of decline slowed as surveys grew longer, with only a 10 percent drop by 40 questions; other research has found that response quality itself degrades toward the end of long surveys. A respondent's profession can also shape which mode works best. One study found that pharmacists sometimes preferred fax, since they routinely receive faxed prescriptions at work but may not always see general mail.

• 7. The Interviewer Effect

  ›

  Race, gender and relative body weight are the interviewer traits most consistently shown to shape how survey respondents answer, an effect that grows stronger when the question relates directly to that trait. An interviewer's race has been shown to affect responses to questions about racial attitudes, an interviewer's sex affects answers on gender-related questions, and an interviewer's body mass index affects answers about eating and dieting. These interviewer effects were first documented mainly in face-to-face surveys, but they also turn up in modes with no visual contact at all, including telephone interviews and video-enhanced web surveys. The usual explanation is social desirability bias, the tendency of participants to present themselves in a way that matches the norms they assume the interviewer holds; researchers treat interviewer effects as one specific case of a broader category called survey response effects, where something about how a question is delivered, not just its content, shapes the answer.

• 8. Where Big Data Fits In

  ›

  Since 2018, survey methodologists have been examining how big data can complement traditional survey methods, drawn to its low cost per data point and to analysis techniques borrowed from machine learning and data mining. That data comes from diverse new sources, including administrative registers, social media, apps and other forms of digital information. Three Big Data Meets Survey Science conferences have been held so far, in 2018-2020 and 2023, with another planned for 2025, alongside dedicated special issues in the Social Science Computer Review, the Journal of the Royal Statistical Society and EPJ Data Science. A book on the subject, Big Data Meets Social Sciences, was edited by Craig A. Hill along with five other Fellows of the American Statistical Association.