Neural network (machine learning)

• 1. Introduction

  ›

  Neural networks have become the engine behind voice assistants, cancer diagnoses, deepfake videos, and the large language models now reshaping how people interact with machines. But the idea behind them is older, stranger, and more biologically grounded than most people realize. At its core, a neural network is a computational model inspired by the structure of the brain itself. Neurons, synapses, signals, weights: the vocabulary of neuroscience was consciously transplanted into mathematics and software. How did that transplant take hold? What kept it from working for decades? And what finally unlocked its extraordinary power?

• 2. Borrowed from Biology

  ›

  Warren McCulloch and Walter Pitts, in 1943, published the first mathematical model of an artificial neuron capable of representing logical functions. Their paper considered neural networks that contain cycles and noted that the current activity of such networks could be affected by activity indefinitely far in the past. That single observation planted the seed for what would later become recurrent networks.

  The connection to biological structure was not merely metaphorical. Each artificial neuron receives numerical inputs, weights them, sums them with a bias term, and then passes the result through a nonlinear activation function to produce an output. The edges connecting neurons model the synapses in the brain. The layered structure, with an input layer, hidden layers, and an output layer, mirrors the way biological neural circuits process information from the sensory periphery inward.

  D. O. Hebb, in the late 1940s, proposed a learning hypothesis based on neural plasticity. That principle, known as Hebbian learning, held that connections between neurons strengthen when those neurons fire together. It became the basis for early experiments including Rosenblatt's perceptron and the Hopfield network. The brain's architecture was not just an inspiration; it was an active blueprint that researchers kept returning to, even as the field grew increasingly mathematical.

• 3. The Perceptron and Its Limits

  ›

  Frank Rosenblatt, a psychologist, described the perceptron in 1958. It was one of the first implemented neural networks and was funded by the United States Office of Naval Research. The announcement triggered public excitement, and the US government drastically increased funding for the field. Researchers spoke optimistically about perceptrons eventually emulating human intelligence, a period later called "the Golden Age of AI".

  That confidence collapsed. Marvin Minsky and Seymour Papert published a book called Perceptrons that highlighted a fundamental flaw: single-layer perceptrons could only solve linearly separable problems. They specifically emphasized that perceptrons were incapable of processing the exclusive-or circuit. The book deflated interest through the late 1960s and into the 1970s.

  What the critics missed is that work on deeper networks had already been underway. Alexey Ivakhnenko and Valentin Lapa published the first working deep learning algorithm in the Soviet Union in 1965, a method they regarded as a form of polynomial regression generalizing Rosenblatt's perceptron. A 1971 paper described a network with eight layers trained by this method. Shun'ichi Amari published the first deep learning multilayer perceptron trained by stochastic gradient descent in 1967. The Minsky-Papert critique was simply irrelevant to these deeper architectures, but the damage to funding and public confidence had already been done.

• 4. Backpropagation Revives the Field

  ›

  Interest in neural networks revived during the 1980s because of a new algorithm: backpropagation. The algorithm works by propagating error gradients backward from the output layer through to the input layer, adjusting the weights at each step. This made it possible to train multi-layer networks efficiently for the first time.

  The history of the algorithm is tangled. Henry J. Kelley developed a precursor in 1960 within control theory. Seppo Linnainmaa published the modern form in his master's thesis in 1970. G. M. Ostrovski et al. republished it in 1971. Paul Werbos applied it explicitly to neural networks in 1982. David E. Rumelhart and colleagues popularized it in 1986 but did not cite the original work. The terminology "back-propagating errors" had actually been introduced by Rosenblatt himself as early as 1962, though he did not describe how to implement it. Backpropagation is mathematically an efficient application of the chain rule, which Gottfried Wilhelm Leibniz derived in 1673.

  Once widely adopted, backpropagation opened the door to training the multi-layer architectures that single-layer critics had dismissed. It also set the stage for the GPU-accelerated training pipelines that would follow. From 1991 to 2015, computing power delivered by GPUs increased around a million-fold, enabling backpropagation to scale to networks of previously unimaginable depth.

• 5. Architectures That Changed Everything

  ›

  Kunihiko Fukushima introduced the neocognitron in 1979, a deep convolutional architecture with convolutional layers, downsampling layers, weight replication, and max pooling. That structure became the foundation for convolutional neural networks, which proved transformative for computer vision.

  Yann LeCun and colleagues built LeNet in 1989 to recognize handwritten ZIP codes on mail; training required three days. By 1998, LeNet-5, a seven-level convolutional network, was being applied by banks to recognize numbers on checks digitized in 32 by 32 pixel images. In October 2012, AlexNet, built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, won the large-scale ImageNet competition by a significant margin over shallow machine learning methods. In 2011, DanNet had already achieved superhuman performance in a visual pattern recognition contest, outperforming traditional methods by a factor of three.

  Long short-term memory, or LSTM, addressed a different problem. Sepp Hochreiter's diploma thesis in 1991 identified the vanishing gradient problem that crippled deep recurrent networks. He and Jürgen Schmidhuber introduced LSTM, which set accuracy records across multiple domains. The forget gate, added in 1999, completed the architecture and made it the default choice for recurrent network design for years.

  Then in 2017, the paper "Attention Is All You Need" introduced the transformer architecture, which uses attention mechanisms to model long-range dependencies in data. Modern large language models including GPT, Gemini, Grok, DeepSeek, and Qwen are all built on this foundation.

• 6. How Training Actually Works

  ›

  Training a neural network means adjusting the weights of its connections until the network's outputs match desired targets closely enough to be useful. The process relies on a loss function that measures the degree of error between what the network predicts and what it should predict. As long as the loss function continues to decline, the network is improving. Training ends when additional observations no longer usefully reduce the cost.

  A hyperparameter is any configurable part of the network whose value is set before training begins: learning rate, batch size, number of layers, and number of nodes are all examples. The learning rate controls how large each corrective step is. A high learning rate shortens training time but sacrifices accuracy; a lower rate takes longer but allows greater precision. Adaptive learning rates, which increase or decrease as appropriate, help avoid oscillating between high and low weight values.

  Networks face a structural hazard called overfitting: when the network is so large relative to the training data that it memorizes the training examples rather than learning to generalize. Cross-validation and regularization techniques address this. In probabilistic frameworks, selecting a larger prior probability over simpler models reduces the tendency to overfit. Neural networks also require far more sample inputs than biological brains need to reach a given level of function. As of 2026, training a commercial large language model typically required hundreds of thousands of computers and cost tens of millions of dollars.

• 7. Where Neural Networks Work

  ›

  From 1988 onward, neural networks transformed the field of protein structure prediction, particularly when cascading networks began training on profiles produced by multiple sequence alignments. That application arrived well before the modern deep learning wave.

  In medicine, neural networks have been used to diagnose cancers and to distinguish highly invasive cancer cell lines from less invasive ones using only cell shape data. They have accelerated reliability analysis of infrastructure subject to natural disasters and predict settling in building foundations. In cybersecurity, they classify Android malware, identify domains belonging to threat actors, and detect URLs posing a security risk.

  Generative adversarial networks, introduced by Ian Goodfellow and colleagues in 2014, drove state-of-the-art image generation through 2018. Nvidia's StyleGAN, released in 2018 and based on the Progressive GAN by Tero Karras and colleagues, achieved excellent image quality and provoked widespread discussion about deepfakes. Diffusion models, emerging in 2015, eventually surpassed GANs in generative modeling, with systems such as DALL-E 2 and Stable Diffusion both arriving in 2022.

  DALL-E was trained on 650 million pairs of images and texts and can create artworks from user text descriptions. Companies including AIVA and Jukedeck have applied transformer architectures to generate original music. Film production companies have used neural networks to analyze the likely financial success of a film. In 2018, Amazon scrapped a recruiting tool after discovering the model favored men over women for software engineering roles because the training data itself reflected a workforce that was predominantly male; the system penalized resumes containing the word "woman" or the name of a women's college.

• 8. Power, Limits, and Open Questions

  ›

  The multilayer perceptron is a universal function approximator, as established by the universal approximation theorem. A recurrent architecture with rational-valued weights can match the power of a universal Turing machine using a finite number of neurons and linear connections. Using irrational values for weights produces a machine with what researchers describe as super-Turing power.

  Despite this theoretical reach, neural networks face real constraints. The brain operates on roughly 20 watts of power. Training commercial transformer models in 2026 required data centers drawing hundreds of megawatts. The statistical properties of real-world data shift over time, a phenomenon called concept drift, which can cause a network's accuracy in deployment to diverge substantially from what was measured during training. Neural networks are also "black box" systems: their internal decision-making processes remain difficult to interpret, and they are vulnerable to adversarial examples that can cause incorrect predictions through deliberately crafted inputs.

  These concerns have spurred research into explainable artificial intelligence and hybrid models that combine neural learning with symbolic reasoning. Neuromorphic engineering has pursued a different path entirely, building non-von-Neumann chips designed to implement neural networks directly in circuitry; Alphabet introduced custom chips called Tensor Processing Units as one example of this direction. The vanishing gradient problem that Hochreiter identified in 1991 took years to solve; the interpretability problem has not yet found its equivalent breakthrough.