Deep learning

• 1. Introduction

  ›

  Deep learning is the technology behind the voice assistant on your phone, the system that reads handwritten checks at the bank, and the software that beat the world champion at Go. At its core, it is a method of teaching machines to learn from raw data by stacking layers of artificial neurons, one atop another, until the system can recognize patterns that no human programmer could have written down by hand. The term itself was introduced to the machine learning community by Rina Dechter in 1986, and it arrived at a moment when researchers were already wrestling with a problem they could barely name: how do you get a machine to understand a face, a word, or a disease without telling it exactly what to look for? That question would take decades to answer. Along the way, it would produce a 2018 Turing Award for Yoshua Bengio, Geoffrey Hinton, and Yann LeCun, an estimated 300,000-fold increase in the computation powering the field's largest projects, and a quiet revolution in how machines perceive the world.

• 2. Roots and Early Pioneers

  ›

  Alexey Ivakhnenko and Lapa published what is now recognized as the first working deep learning algorithm in 1965. Their method, called the Group method of data handling, could train arbitrarily deep neural networks by building up layers through regression analysis and pruning unnecessary units with a separate validation set. A 1971 paper from the same lineage described a network with eight layers. That same year, Kaoru Nakano published other early recurrent neural network work, and in 1972 Shun'ichi Amari made an earlier architecture adaptive. The first deep learning multilayer perceptron trained by stochastic gradient descent was also Amari's, published in 1967, where a five-layer network learned to classify patterns that were not linearly separable.

  Frank Rosenblatt's 1958 perceptron had proposed a three-layer network, but the hidden layer held only randomized weights that did not learn. His 1962 book went further, describing a four-layer system with adaptive weights and crediting H. D. Block and B. W. Knight, while citing an earlier network by R. D. Joseph from 1960 as functionally equivalent. The learning algorithm in Joseph's design, however, was not functional and slipped into obscurity.

  Backpropagation, the training technique that would eventually dominate the field, has an equally tangled genealogy. Gottfried Wilhelm Leibniz derived the underlying chain rule in 1673. Henry J. Kelley worked out a continuous precursor in 1960. The modern form first appeared in Seppo Linnainmaa's master thesis in 1970, was republished by G. M. Ostrovski and colleagues in 1971, and was applied to neural networks by Paul Werbos in 1982. David E. Rumelhart and colleagues popularized it in 1986 without citing the original work. In 1969, Kunihiko Fukushima introduced the rectified linear unit, which would later become the most widely used activation function in the field.

• 3. The Long Winter

  ›

  Through the 1990s and into the 2000s, neural networks fell out of favor. Simpler models built on handcrafted features, such as Gabor filters and support vector machines, dominated research because neural networks were too expensive to run and too poorly understood theoretically. Researchers working on speech recognition largely abandoned the approach in favor of Gaussian mixture models and Hidden Markov models, which were better understood and more practically manageable.

  An exception appeared at SRI International in the late 1990s, funded by the NSA and DARPA. The speaker recognition team led by Larry Heck reported meaningful success with deep neural networks in the 1998 NIST Speaker Recognition benchmark. That work was deployed in the Nuance Verifier, marking the first major industrial use of deep learning.

  LSTM, the long short-term memory architecture published in 1995, took years to prove itself. It grew from Sepp Hochreiter's 1991 diploma thesis, which identified the vanishing gradient problem and proposed recurrent residual connections as a fix. The architecture was not yet complete; it required a forget gate, introduced in 1999, to reach the form that became standard. By 2003, LSTM was finally competitive with traditional speech recognizers on certain tasks. In 2006, Alex Graves, Santiago Fernandez, Faustino Gomez, and Schmidhuber combined LSTM with connectionist temporal classification in stacked architectures. In 2009, the approach became the first recurrent neural network to win a pattern recognition contest, in connected handwriting recognition.

• 4. The GPU Breakthrough

  ›

  In 2009, Raina, Madhavan, and Andrew Ng trained a network of 100 million parameters on 30 Nvidia GeForce GTX 280 GPUs, reporting training speeds up to 70 times faster than CPU-based methods. That demonstration pointed toward a hardware path that would transform the field. By 2019, graphics processing units with AI-specific enhancements had displaced CPUs as the standard for training large commercial cloud models. OpenAI measured the computation used across the largest deep learning projects from AlexNet in 2012 to AlphaZero in 2017 and found a 300,000-fold increase, with a doubling time of roughly 3.4 months.

  In 2011, a convolutional network called DanNet, built by Dan Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, and Jurgen Schmidhuber, achieved superhuman performance in a visual pattern recognition contest for the first time, outperforming traditional methods by a factor of three. October 2012 brought another turning point: AlexNet, designed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, won the large-scale ImageNet competition by a margin that stunned the computer vision community. Further advances came from VGG-16, built by Karen Simonyan and Andrew Zisserman, and Google's Inceptionv3.

  The degradation problem, where stacking too many layers caused training accuracy to fall rather than rise, was addressed in 2015 by two separate techniques. The highway network appeared in May of that year, and the residual neural network, ResNet, followed in December. ResNet behaves, as the source notes, like an open-gated highway network. That same year, Google's speech recognition system improved by 49% through an LSTM-based model, deployed through Google Voice Search on smartphones.

• 5. How Machines Learn to See and Hear

  ›

  Yann LeCun and colleagues built LeNet in 1989 for the specific task of reading handwritten ZIP codes on mail. Training the network took three days. By 1998, a seven-level version called LeNet-5 was being used by several banks to read hand-written numbers on checks digitized in 32-by-32 pixel images. LeCun later estimated that by the early 2000s, CNNs were already processing somewhere between 10% and 20% of all checks written in the United States.

  Speech recognition took a parallel path. The TIMIT dataset, containing 630 speakers reading across eight major dialects of American English, became the standard proving ground. Error rates fell steadily as architectures improved: early random recurrent networks showed phone error rates around 26%, while later hierarchical convolutional deep maxout networks brought that figure down to 16.5%. By the time of the 2009 NIST workshop on deep learning for speech, researchers found that replacing pre-training with large amounts of data and direct backpropagation produced error rates dramatically lower than the best Gaussian mixture model systems. All major commercial speech recognition systems, including Microsoft Cortana, Amazon Alexa, Apple Siri, and Google Now, now rest on deep learning foundations.

  In 2012, Andrew Ng and Jeff Dean trained a network that learned to identify higher-level concepts, including cats, using only unlabeled images pulled from YouTube. No human had labeled the images; the network found the structure on its own. That demonstration of unsupervised learning at scale captured something that handcrafted feature engineering had never managed.

• 6. Beyond Vision: Medicine, Science, Language

  ›

  AtomNet, a deep learning system for structure-based drug design, was used to predict candidate molecules for disease targets including the Ebola virus and multiple sclerosis. In 2017, graph neural networks were applied for the first time to predict properties of molecules in a large toxicology dataset. By 2019, generative neural networks produced molecules that were validated all the way through mouse experiments.

  In bioinformatics, the most striking result came in 2020, when AlphaFold achieved an accuracy in protein structure prediction significantly higher than all previous computational approaches. Deep neural networks trained on more than 6,000 blood samples produced an epigenetic aging clock drawing on information from 1,000 CpG sites, with plans for public release in 2021 through an Insilico Medicine spinoff called Deep Longevity.

  In materials science, Google DeepMind and Lawrence Berkeley National Laboratory announced in November 2023 that a system called GNoME had discovered over two million new materials. Autonomous robotic experiments validated the predictions at a success rate of 71%. The data was made publicly available through the Materials Project database.

  Natural language processing was transformed by LSTM and later by transformers. Google Translate, which supports over one hundred languages, uses an end-to-end LSTM network that encodes the semantics of a full sentence rather than matching phrase by phrase. The system uses English as an intermediate between most language pairs. The pre-training approach at the heart of large language models traces directly to Jurgen Schmidhuber's 1991 neural history compressor, and, as the source notes, the letter P in ChatGPT refers to that same pre-training concept.

• 7. Shadows and Vulnerabilities

  ›

  In 2013, researchers showed that small, carefully designed perturbations to images that a network classified correctly could cause confident misclassification. A year later, networks were shown to classify clearly unrecognizable images as belonging to familiar categories. These behaviors led Ben Goertzel to hypothesize that the failures reflect fundamental limitations in the internal representations deep networks form.

  Adversarial attacks have grown more varied. In 2016, one group used an artificial neural network to modify images in trial-and-error fashion, identifying another network's focal points and generating images that fooled it, while looking unchanged to human eyes. Separately, printouts of doctored images, when photographed, successfully tricked image classification systems. In 2017, researchers added stickers to stop signs and caused a network to misclassify them. Another experiment showed that certain psychedelic spectacles could fool facial recognition into identifying ordinary people as celebrities. Also in 2016, specific sounds were demonstrated to make the Google Now voice command system open a designated web address.

  Data poisoning, where false data is fed continuously into a training set, represents a slower form of attack. The training data itself carries its own ethical questions. Philosopher Rainer Muhlhoff identified five forms of what he calls machinic capture of human micro-work: gamification, trapping and tracking through tools like CAPTCHAs, exploitation of social motivations such as face tagging on social platforms, information mining via wearable devices, and direct clickwork. The low-paid labor involved in annotating training data has attracted sustained criticism alongside the technical vulnerabilities the systems themselves carry.