Regression analysis

• 1. Introduction

  ›

  Regression analysis begins with a question that has driven scientists for centuries: when one thing changes, what happens to something else? Isaac Newton encountered this problem in 1700 while studying equinoxes, earning a credit historians describe as "an embryonic linear regression analysis." Newton had already used a related averaging method in his 1671 work on Newton's rings, well before such techniques had a name. Today, regression analysis underlies prediction and forecasting across virtually every field of quantitative study. How did a method born in astronomy become one of the most broadly applied tools in science? What does it actually do, and what can go wrong when its assumptions crack?

• 2. Naming the Regression

  ›

  Francis Galton gave regression analysis its name in the 19th century, though the phenomenon he was describing was biological, not mathematical. Galton noticed that the heights of descendants of unusually tall ancestors tend to drift back toward a normal average, a pattern he called regression toward the mean. His idea carried no broader statistical ambition at first. Udny Yule and Karl Pearson later extended Galton's biological observation into a general statistical framework. In their formulation, both the response variable and the explanatory variables were assumed to follow a joint Gaussian distribution. R. A. Fisher narrowed that assumption in his works of 1922 and 1925, arguing only that the conditional distribution of the response variable needed to be Gaussian, not the joint distribution. Fisher's weaker requirement brought the theory closer in spirit to Gauss's own formulation from 1821.

• 3. Legendre, Gauss, and Least Squares

  ›

  Adrien-Marie Legendre published the method of least squares in 1805; Carl Friedrich Gauss published his own account in 1809. Both men applied the technique to the same class of problem: determining the orbits of bodies around the Sun from astronomical observations. Their subjects were mostly comets, and later the newly discovered minor planets. Gauss did not stop there. In 1821 he published a further development of the theory, which included an early form of what is now called the Gauss-Markov theorem, a result that identifies conditions under which least squares estimates are optimally efficient among all linear unbiased estimators. That theorem remains a cornerstone justification for why ordinary least squares is so widely used.

• 4. What Regression Actually Does

  ›

  A regression model estimates the relationship between a dependent variable and one or more independent variables. Ordinary least squares, the most common approach, finds the line or hyperplane that minimizes the sum of squared differences between the observed data points and the fitted values. For any estimated model, the residual for each observation is simply the difference between the value the model predicts and the value actually recorded in the data. To estimate a least squares model with a given number of distinct parameters, a researcher must have at least that many distinct data points; otherwise the system is underdetermined and infinitely many solutions fit equally well. The independent variables must also be linearly independent of one another, a condition that ensures a unique solution exists.

• 5. Two Purposes, One Tool

  ›

  Regression serves two conceptually separate purposes that are easy to conflate. Prediction and forecasting form one purpose, with extensive overlap with machine learning practice. The second purpose is inferring causal relationships between variables. A regression by itself reveals only associations within a fixed dataset. Justifying either use requires additional argument from the researcher: predictive power in a new context is not automatic, and causal interpretation from observational data is especially demanding. Extrapolation beyond the range of data used to fit the model amplifies the risk further, because prediction intervals tend to expand rapidly as the independent variables move outside the observed range. The further the extrapolation, the more the result depends on assumptions rather than evidence.

• 6. When Assumptions Bend

  ›

  Classical regression assumptions include a representative sample, error-free measurement of independent variables, residuals with an expected value of zero, constant variance across observations, and uncorrelated errors. In practice these conditions rarely hold exactly. Econometrics as a subfield is largely organized around developing techniques that preserve useful properties when real data violates the classical framework. Heteroscedasticity-consistent standard errors allow the variance of residuals to change across values of the independent variable. Clustered standard errors, geographic weighted regression, and Newey-West standard errors each address correlated error patterns within subsets of data. When independent variables are measured with error, the errors-in-variables model provides a pathway to reasonable estimates.

• 7. Beyond Straight Lines

  ›

  Linear regression does not require that the relationship between variables be literally straight. Adding a squared term to the independent variable produces a parabola, yet the model remains linear regression because the parameters enter the equation linearly. Nonlinear regression, by contrast, involves model functions that are not linear in the parameters, and fitting them requires iterative numerical procedures rather than a closed-form solution. Limited dependent variables introduce a further class of complications. When the response variable is binary, the linear probability model applies ordinary least squares directly, while the probit and logit models offer nonlinear alternatives. For categorical responses with more than two values, the multinomial logit applies; for ordered categories, ordered logit and ordered probit models exist. When the dependent variable is observed only under certain conditions, censored regression models or Heckman correction methods can account for the resulting selection problem.

• 8. Computing Power and Practice

  ›

  In the 1950s and 1960s, economists calculated regressions using electromechanical desk calculators. Before 1970, a single regression could take up to 24 hours to complete. Modern regression is carried out on computers and even handheld scientific and graphing calculators using statistical and spreadsheet software packages. Not all techniques are equally standardized across software: simple and multiple linear regression are available in most packages and even some spreadsheet applications, while nonparametric and robust methods vary in implementation and may carry the same name while computing different things. Specialized regression software has also been developed for particular domains, including survey analysis and neuroimaging, reflecting how far the method has traveled from Legendre and Gauss tracking comets across the sky.