Sepp Hochreiter submitted a German diploma thesis in 1991 that identified the critical flaw known as the vanishing gradient problem. His supervisor Jürgen Schmidhuber considered this identification of the barrier highly significant for recurrent neural networks.
An LSTM unit typically consists of a cell and three specific gates: an input gate, an output gate, and a forget gate. These gates regulate information flow by mapping values between zero and one to decide what to retain or discard from previous states.
Felix Gers, Jürgen Schmidhuber, and Fred Cummins introduced the forget gate into the architecture in 1999. This addition enabled the LSTM to reset its own state and deal with the vanishing gradient problem effectively.
Google started using an LSTM trained by CTC for speech recognition on Google Voice in 2015. The new model cut transcription errors by 49 percent according to their official blog post.
Facebook performed some 4.5 billion automatic translations every day using long short-term memory networks in 2017. This usage demonstrated the scale at which the technology could handle translation tasks during that year.