Short answers, pulled from the story.
Gradient descent is generally attributed to Augustin-Louis Cauchy, who first suggested it in 1847. Jacques Hadamard independently proposed a similar method in 1907, and Haskell Curry first studied its convergence properties for non-linear optimization problems in 1944.
Gradient descent is used to minimize the cost or loss function in machine learning and artificial intelligence. Stochastic gradient descent, a direct extension of the method, is described as the most basic algorithm used for training most deep networks today.
Standard gradient descent computes updates using the full objective function, while stochastic gradient descent adds a stochastic property to the update direction, typically drawing on a subset of data per step. Both methods underlie the backpropagation algorithms used to train artificial neural networks.
In steepest descent, successive residual vectors are orthogonal across iterations, which forces the algorithm to alternate between directions aligned with the extreme axes of the function's elongated level sets. This behavior is caused by a high condition number in the system matrix and results in slow, oscillating convergence.
Nesterov acceleration is a modification proposed by Yurii Nesterov that achieves faster convergence for convex problems. For a convex Lipschitz function, the standard method bounds error at a certain rate per step; the Nesterov technique reduces that error at a fundamentally faster rate. A later refinement called the optimized gradient method cuts the constant factor by two.
A step size that is too small causes very slow convergence, while one that is too large leads to overshoot and divergence. Strategies for choosing a suitable step size include the Wolfe conditions, backtracking line search, and the sequence used in the Barzilai-Borwein method.