Short answers, pulled from the story.
Francis Galton coined the term in the 19th century to describe a biological phenomenon: the heights of descendants of tall ancestors tend to regress down toward a normal average. Galton's use was strictly biological; Udny Yule and Karl Pearson later extended it to a general statistical context.
Adrien-Marie Legendre published the method of least squares in 1805, and Carl Friedrich Gauss published his own account in 1809. Both applied it to determining the orbits of comets and minor planets from astronomical observations.
Regression analysis is used for prediction and forecasting, and for inferring causal relationships between variables. Using regression for either purpose requires the researcher to justify why the observed relationships apply to the new context or carry a causal interpretation.
The Gauss-Markov theorem identifies conditions under which least squares estimates are unbiased, consistent, and efficient among all linear unbiased estimators. Gauss published an early form of this result in 1821 as part of a further development of the theory of least squares.
Before 1970, a single regression could take up to 24 hours to complete. Economists in the 1950s and 1960s used electromechanical desk calculators for this work.
Interpolation means predicting within the range of values used to fit the model; extrapolation means predicting outside that range. Extrapolation relies heavily on the regression assumptions, and prediction intervals expand rapidly as the independent variables move beyond the observed data range.