Long short-term memory

• 1. Introduction

  ›

  Long short-term memory, or LSTM, is a piece of artificial intelligence architecture that solved a problem researchers had been stuck on for decades. In the early 1990s, a graduate student in Germany named Sepp Hochreiter was staring at a fundamental flaw in how neural networks learn. When these networks tried to find patterns across long stretches of information, something broke. The learning signal simply vanished before it could travel back through time. Hochreiter's 1991 diploma thesis laid out the problem with unusual precision, and his supervisor, Jürgen Schmidhuber, considered the work highly significant. What followed was a design that became one of the most widely deployed architectures in the history of machine learning. By 2017, Facebook was using LSTM to perform roughly 4.5 billion automatic translations every single day. How did a piece of university research become the engine behind Google Translate, Apple's Siri, Amazon's Alexa, and a record-breaking handwriting recognition system? The answers lie in a cell, three gates, and the elegant notion of a memory that knows when to forget.

• 2. The Vanishing Gradient Problem

  ›

  Classic recurrent neural networks could, in theory, track long-range dependencies in sequences. In practice, training them was a different matter. When a network is trained through a technique called backpropagation, error signals travel backward through each time step. Across long sequences, those signals could shrink toward zero, which effectively stopped the network from learning anything about events far back in time. Hochreiter's 1991 German diploma thesis was the first rigorous analysis of exactly why this happened. The culprit was arithmetic: very small numbers compounding multiplicatively across many steps caused the gradient to simply vanish. His supervisor, Schmidhuber, recognized the thesis as unusually significant for work at that level. The two researchers continued collaborating, and an early version of their proposed solution appeared in a 1995 technical report before being presented at the NIPS 1996 conference. The full paper that became the standard reference was published in 1997 in the journal Neural Computation. That paper introduced what they called the Constant Error Carousel, or CEC, a mechanism that allowed error signals to flow backward without attenuation. That insight sits at the core of the LSTM cell to this day.

• 3. A Cell and Three Gates

  ›

  Felix Gers, Jürgen Schmidhuber, and Fred Cummins added the piece that made the architecture practical. In 1999, they introduced the forget gate, sometimes called the keep gate, which gave the network the ability to reset its own internal state. Before this addition, the original LSTM had no mechanism for actively discarding information it no longer needed. The forget gate maps the previous state and the current input to a value between 0 and 1. A value of 1 means the information is fully retained; a value of 0 means it is discarded. The same team added peephole connections in 2000, allowing each gate to look directly at the cell's internal state rather than only at its output. The sentence-level example the original researchers used to illustrate the design is clarifying: an LSTM processing "Dave, as a result of his controversial claims, is now a pariah" can remember the grammatical gender and number of the subject Dave long enough to correctly associate it with the pronoun his, then discard that information once the verb is appears, because it is no longer needed. The three gates, input, output, and forget, each assign a value between 0 and 1 to each piece of information, regulating what enters, what persists, and what gets released to the next layer.

• 4. Competitions and Breakthroughs

  ›

  In 2009, an LSTM trained by a technique called Connectionist Temporal Classification entered the ICDAR connected handwriting recognition competition. Three models were submitted by a team led by Alex Graves. One of the three was the most accurate model in the entire competition; another was the fastest. It was the first time a recurrent neural network won an international competition of that kind. CTC itself had been introduced three years earlier, in 2006, when Graves, Fernandez, Gomez, and Schmidhuber published the error function as a way to achieve simultaneous alignment and recognition of sequences. In 2013, Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton used LSTM networks as a major component of a system that achieved a record 17.7% phoneme error rate on the TIMIT natural speech dataset. The TIMIT dataset is a classic benchmark in speech research, and the phoneme error rate of 17.7% represented a new floor at the time. The same CTC training method that powered handwriting recognition would later be the mechanism Google chose when it deployed LSTM for speech recognition on Google Voice, cutting transcription errors by 49%.

• 5. Industry Deployment

  ›

  Google began using LSTM trained by CTC for speech recognition on Google Voice in 2015. The same year, Srivastava, Greff, and Schmidhuber used LSTM principles to design the Highway network, a feedforward architecture with hundreds of layers and the deepest networks built to that point. In 2016, Google deployed LSTM inside the Allo conversation app to suggest messages to users, and released the Google Neural Machine Translation system for Google Translate, which used LSTMs to reduce translation errors by 60%. That same year, Apple announced at its Worldwide Developers Conference that it would use LSTM for the QuickType keyboard on iPhone and for Siri. Amazon's Polly system, which generates the voice behind Alexa, used a bidirectional LSTM for its text-to-speech conversion. Microsoft reported reaching 94.9% recognition accuracy on the Switchboard corpus, a demanding benchmark incorporating a vocabulary of 165,000 words, using what it called dialog session-based long-short-term memory. By 2018, OpenAI had trained an LSTM using policy gradients to beat human players in Dota 2 and to control a robot hand capable of manipulating physical objects with precision that had not been achieved before. DeepMind followed in 2019 by using a policy-gradient-trained LSTM to excel at Starcraft II.

• 6. Variants and Evolution

  ›

  In 2005, Graves and Schmidhuber published bidirectional LSTM alongside the full backpropagation-through-time formulation, allowing networks to process sequences in both temporal directions. In 2014, Kyunghyun Cho and others published a simplified variant called the Gated Recurrent Unit, or GRU, which condensed the forget gate LSTM into a leaner design. In 2017, researchers from Michigan State University, IBM Research, and Cornell University presented a time-aware variant called T-LSTM at the Knowledge Discovery and Data Mining conference; that version performed better than standard LSTM on certain datasets where the time intervals between measurements are irregular. In 2020, Kaplan and McCandlish compared the scaling behavior of LSTM and Transformer architectures in language modeling, finding that Transformers scale more efficiently than LSTM on text data. Sepp Hochreiter, the researcher whose 1991 diploma thesis started the whole lineage, later led a team that published a modern upgrade called xLSTM. The architecture contains two block types: one called mLSTM, which is parallelizable in the same way as the Transformer, and another called sLSTM, which preserves the state-tracking capability that has always been the architecture's core advantage.