Short answers, pulled from the story.
A recurrent neural network (RNN) is designed for processing sequential data where the order of elements matters. Unlike feedforward neural networks, which process each input independently, RNNs use recurrent connections that feed the output of a neuron at one time step back as input at the next, maintaining a hidden state that captures temporal patterns across the sequence.
Hochreiter and Schmidhuber invented LSTM in 1995 to solve the vanishing gradient problem in traditional RNNs. The vanishing gradient problem prevented networks from learning long-range dependencies, because error signals shrank exponentially with the distance between an event and the weight being corrected. LSTM uses recurrent gates, including a forget gate, to allow errors to flow backward through effectively unlimited virtual layers.
Recurrent neural networks have been applied to speech recognition, machine translation, handwriting recognition, text-to-speech synthesis, music composition, robot control, time series anomaly detection, energy forecasting, protein homology detection, brain-computer interfaces, and prediction of plasma disruptions in fusion reactors. Bidirectional LSTM was used in Google voice search and Android dictation.
The vanishing gradient problem is the tendency of error signals to shrink exponentially as they are propagated backward through many time steps during training. This makes it difficult for a standard RNN to learn patterns that depend on events separated by a large number of time steps. LSTM and GRU architectures were specifically designed to address this limitation.
Gated Recurrent Units were introduced in 2014 as a simplified alternative to LSTM. GRUs have fewer parameters because they lack an output gate, making them more computationally efficient. Performance on tasks like polyphonic music modeling and speech signal modeling was found to be similar to LSTM, with no consistent advantage identified for either architecture.
The concept traces to two parallel lines of work in the early twentieth century. In 1901, Cajal observed recurrent loop structures in the cerebellar cortex, and in 1933, Lorente de Nó described recurrent reciprocal connections using Golgi's staining method. Separately, the Ising model of the 1920s and Hopfield's 1982 paper applied statistical mechanics to network memory. Frank Rosenblatt published close-loop cross-coupled perceptrons in 1960, describing recurrent connections in three-layer networks.