Meta-analysis

• 1. Introduction

  ›

  Meta-analysis is a method for synthesizing quantitative findings from multiple independent studies that address the same research question. In 1904, the statistician Karl Pearson did something that would look familiar to any modern researcher. He gathered results from several separate studies on typhoid inoculation and pooled them into a single analysis, published in the British Medical Journal. That act of combination, of treating many small answers as pieces of a larger puzzle, planted the seed of a methodology that would eventually become one of the most contested and consequential tools in science.

  How does a researcher make sense of dozens of studies that all asked the same question but reached different conclusions? And who decides which studies are even worth combining? Those are the questions that meta-analysis was built to answer, and they turn out to be far harder than they first appear.

• 2. Naming the Method

  ›

  Gene V. Glass, a statistician, coined the term "meta-analysis" in 1976. Writing in the journal Educational Researcher, he defined it plainly: "meta-analysis refers to the analysis of analyses." His goal was to describe aggregated measures of relationships and effects, not just to collect studies but to extract meaning from their combined weight.

  The first meta-analysis that followed Glass's framework closely was published in 1978, authored by Mary Lee Smith and Glass. It examined the effectiveness of psychotherapy. The response from some corners of the scientific community was blunt. Hans Eysenck, in a 1978 article directly responding to Smith and Glass, called the method an "exercise in mega-silliness." He would later reach for an even sharper dismissal, labeling it "statistical alchemy." Despite that reception, the field grew. By 1991, there were 334 published meta-analyses. By 2014, that number had risen to 9,135.

• 3. Building the Dataset

  ›

  Before a meta-analyst can calculate anything, the data must be found. Researchers typically search multiple databases, with PubMed, Embase, and PsycINFO among the most widely used. Boolean operators and search limits help narrow results. Many scientists run duplicate search terms across two or more databases to catch what a single source might miss.

  A technique called snowballing extends that reach further. Researchers comb the reference lists of eligible studies, hunting for additional sources the original search missed. Every decision along the way, how many studies were returned, how many were discarded, and why, must be logged in a document called a PRISMA flow diagram. The date range covered by the search, and the date on which the search itself was conducted, must both be recorded.

  Grey literature introduces its own complications. This category includes conference abstracts, dissertations, and pre-prints, work that was never formally published. Including it reduces the risk of skewing the final result toward studies that made it to print. The trade-off is quality: reports from conference proceedings, the most common source of grey literature, are poorly reported, and data in subsequent publications differ from conference versions in almost 20% of cases.

• 4. Models and Their Tensions

  ›

  Once the data are collected, researchers must choose a statistical model. The fixed effect model treats all included studies as if they investigated the same population under identical conditions. It assigns each study a weight based on the inverse of its variance, meaning larger studies dominate the result. Critics note that this assumption is typically unrealistic, since studies rarely replicate conditions exactly.

  The random effects model attempts to account for variability between studies, what statisticians call heterogeneity. It applies a random effects variance component to adjust each study's weight. As heterogeneity increases, the model redistributes weight away from larger studies and toward smaller ones, eventually arriving at a simple unweighted average. One persistent critique is that the confidence intervals produced by this model tend to underestimate statistical error, making conclusions appear more certain than the data warrant.

  Doi and Thalib introduced a third approach, the quality effects model, which incorporates an assessment of each study's methodological quality into the weighting. If one study is judged to be more rigorous than others, a portion of the weight assigned to the weaker studies is mathematically redistributed to the stronger one. A recent evaluation found this model outperformed the random effects model on measures of mean squared error and true variance under simulation.

  For researchers comparing multiple treatments simultaneously, network meta-analysis offers another path. The Bucher method works by comparing treatments in closed loops of three, where one treatment serves as the common node. More complex approaches use Bayesian hierarchical models, written as directed acyclic graphs and run through Markov chain Monte Carlo software such as WinBUGS. Software packages including metaBMA and RoBMA have since simplified that process, and the method is now available through graphical interfaces such as JASP.

• 5. The Publication Bias Problem

  ›

  Studies that find no effect, or find results that run against expectations, are less likely to reach publication. That asymmetry has a name: publication bias. The informal description is the file drawer problem. Negative or insignificant results get tucked away rather than submitted, and the published literature drifts toward the positive.

  Pharmaceutical companies have been documented hiding negative trials. Researchers may overlook dissertation studies or conference abstracts that never reached a journal. The scale of the distortion is hard to pin down because, as the source puts it, one cannot know how many studies have gone unreported. Estimates suggest that as many as 25% of meta-analyses in the psychological sciences may have been affected.

  Researchers use a funnel plot to look for signs of this distortion. The plot places effect size on one axis and standard error on the other. Small studies, which carry more uncertainty, scatter widely at the base. Large studies cluster tightly at the tip. If the base is skewed to one side, that asymmetry suggests smaller studies pointing in a favorable direction were published while unfavorable ones were not. Statistical tests for this asymmetry exist, but they carry low statistical power and can produce false positives in some circumstances.

  Questionable research practices add a separate layer of distortion. Reworking statistical models until a significant result appears can inflate the published literature even when no publication bias from editorial decisions is present.

• 6. Agenda and Conflict of Interest

  ›

  A 2011 study examined 29 meta-analyses drawn from general medicine journals, specialty medicine journals, and the Cochrane Database of Systematic Reviews. Together those 29 meta-analyses covered 509 randomized controlled trials. Of the 318 trials that reported funding sources, 219, or 69%, received funding from industry. Of the 509 trials, 132 reported author conflict of interest disclosures, and 91 of those, again 69%, named at least one author with financial ties to industry. Yet only two of the 29 meta-analyses mentioned RCT funding sources, and none reported those financial ties.

  The consequences of agenda-driven analysis can reach beyond the literature. In 1998, a US federal judge found that the United States Environmental Protection Agency had abused the meta-analysis process to produce a study claiming cancer risks to non-smokers from environmental tobacco smoke. The stated intent was to influence policymakers to pass smoke-free-workplace laws. The case illustrated how the flexibility of meta-analysis, its many judgment calls about inclusion, weighting, and framing, can be exploited when the analyst has a predetermined destination in mind.

• 7. What Meta-Analysis Has Become

  ›

  Modern meta-analysis reaches well beyond pooling averages. Researchers can test whether the spread of outcomes across studies exceeds what random sampling alone would predict. They can code study characteristics, the measurement instrument used, the population sampled, the design choices made, and use those codes to reduce variance in the overall estimate.

  The method has moved into genomics, where whole genome sequencing studies are combined to detect rare variants linked to complex traits. Neuroimaging research uses seed-based d mapping, a technique that meta-analyzes differences in brain activity measured by fMRI, VBM, or PET scans. MicroRNA expression profiles have been pooled across studies to identify gene activity patterns across different tissue types and disease conditions.

  In applied behavioral science, researchers have proposed large-scale studies called megastudies, which test many different interventions simultaneously using a single platform. One such study recruited participants through a fitness chain. The proposal addresses a persistent complaint about behavioral meta-analyses: that "different scientists test different intervention ideas in different samples using different outcomes over different time intervals," making individual studies difficult to compare and limiting their usefulness for policy.

  The push for open practices in science has also reached this field. Tools for crowd-sourced living meta-analyses, updated continuously by communities of scientists, aim to make the subjective choices involved more visible and more open to scrutiny. Software platforms for running these analyses now include R packages such as metafor and meta, along with RevMan, JASP, Jamovi, StatsDirect, MetaEssential, and Comprehensive Meta-Analysis.