Feedforward neural network

• 1. Introduction

  ›

  Feedforward neural networks sit at the heart of modern artificial intelligence, yet their story stretches back to a problem that has nothing to do with computers. Around 1800, Adrien-Marie Legendre and Carl Friedrich Gauss built what historians now recognize as the simplest version of a feedforward network: a single layer of weights trained by the least squares method. They used it to predict the movements of planets from observational data. The technique they relied on, minimizing mean squared error, is better known today as linear regression.

  What makes a feedforward network distinct is deceptively simple. Information travels in one direction only, from inputs through weights to outputs. There are no loops, no feedback, no signal doubling back on itself. That constraint turns out to be not a limitation but a design requirement. Loops create an infinite regress that cannot be solved by the learning algorithm at the core of modern neural networks. The question of why that matters, and how researchers spent nearly two centuries learning to exploit it, is what this documentary explores.

  Along the way, credit for key breakthroughs was disputed, delayed, or simply lost. A Finnish student published a foundational algorithm in a master thesis that the field largely ignored for years. Two researchers at MIT Lincoln Laboratory may have beaten Frank Rosenblatt to an important invention, then walked away. And an eight-layer neural network trained in 1971 by a Soviet scientist anticipated techniques that would not become mainstream for decades.

• 2. How Information Flows

  ›

  Warren McCulloch and Walter Pitts, in 1943, proposed the binary artificial neuron as a logical model of biological neural networks. Their model was spare and abstract: a neuron either fires or it does not. But even that minimal architecture encoded the directional logic that defines feedforward computation. Inputs arrive, a decision is made, an output is produced. Nothing flows backward.

  The mathematical machinery behind that flow centers on activation functions. Historically, two sigmoid-shaped functions dominated. One is the hyperbolic tangent, which ranges from -1 to 1. The other is the logistic function, similar in shape but ranging from 0 to 1. In both cases, the output of a given node depends on the weighted sum of its input connections. In recent deep learning practice, the rectified linear unit, known as ReLU, has largely displaced the sigmoids. The shift was driven by numerical problems that the sigmoid functions caused during training, problems that ReLU sidesteps.

  There is a naming confusion worth flagging. Some computer scientists and researchers in other fields who study biological brain networks use the term feedforward differently, or apply it inconsistently. The source of that confusion is the fact that the word describes an architectural property, not a specific model, and different communities imported it with slightly different meanings.

• 3. Learning Through Error

  ›

  Frank Rosenblatt proposed the multilayered perceptron model in 1958. His design included an input layer, a hidden layer with randomized weights that did not learn, and an output layer with connections that could be adjusted. The model pointed toward something important but left a critical problem unsolved: no one yet had a working algorithm for training the hidden layers.

  Learning in a feedforward network works by comparing the network's output against the desired result, measuring the gap, and adjusting the connection weights to close it. This is supervised learning. The adjustment procedure is called backpropagation, and it relies on gradient descent: weights shift in the direction that reduces error most efficiently. A value called the learning rate controls how large each step is, balancing speed of convergence against the risk of overshooting.

  The derivative required to update a hidden layer node is more complex than the one required for an output node. The calculation works backward through the layers, using the derivative of the activation function at each step. That chain of derivatives is why the algorithm is called backpropagation: the signal indicating how wrong the network was propagates backward through the network to assign responsibility to each weight.

  Seppo Linnainmaa published the modern form of backpropagation in his master thesis in 1970. G. M. Ostrovski and colleagues republished the result in 1971. Paul Werbos applied backpropagation specifically to neural networks in 1982, drawing on his 1974 PhD thesis. David E. Rumelhart and colleagues popularized the algorithm in 1986, though they did not cite the earlier work.

• 4. Buried Discoveries

  ›

  R. D. Joseph, writing in 1960, named Farley and Clark of MIT Lincoln Laboratory as researchers who had actually preceded Rosenblatt in developing a perceptron-like device. Joseph's account adds a deflating coda: they dropped the subject. An early lead, apparently abandoned.

  Alexey Grigorevich Ivakhnenko and Valentin Lapa published something far more consequential in 1965: the Group Method of Data Handling, which the source identifies as the first working deep learning algorithm. Their method trained arbitrarily deep neural networks layer by layer, using regression analysis at each stage. Superfluous hidden units were pruned away using a separate validation set. The activation functions in their network were Kolmogorov-Gabor polynomials, which are multiplicative rather than additive, making their 1965 model the first deep network to use what would later be called multiplicative units or gates. They demonstrated the approach on an eight-layer network trained in 1971.

  Shun'ichi Amari reported the first multilayered neural network trained by stochastic gradient descent in 1967. His network could classify pattern classes that were not linearly separable, a significant capability. The computer experiments were conducted by Amari's student Saito, using a five-layered feedforward network with two learning layers.

  Interest in backpropagation networks faded and then returned. In 2003, Yoshua Bengio and co-authors achieved successes applying deep learning to language modelling, and the field rediscovered what had been accumulating in the literature for decades.

• 5. Perceptrons and Their Limits

  ›

  A perceptron is what you get when you apply a linear threshold function as the activation. The term is sometimes used to mean a single such unit, sometimes to mean a small network of them. The perceptron has a simple training rule called the delta rule, which calculates the difference between actual and expected output and adjusts weights accordingly. That is gradient descent in a minimal form.

  Single perceptrons have a well-known ceiling: they can only separate classes that are linearly separable in the input space. Arrange the examples so no straight line divides them, and a single perceptron fails. Multiple parallel nonlinear units can escape that limit. They are collectively able to approximate any continuous function from a compact interval of the real numbers to the interval -1 to 1, even though each individual unit with a linear threshold is computationally limited.

  The multilayer perceptron, or MLP, addresses the separation problem directly by stacking layers. The term is technically a misnomer for the modern feedforward network, because modern MLPs use nonlinear activation functions rather than simple thresholds. The architecture requires at least three layers and is notable precisely for its ability to handle data that is not linearly separable. A synonym sometimes used is fully connected network, reflecting the fact that every neuron in one layer connects to every neuron in the next.

• 6. Variants and Extensions

  ›

  Convolutional neural networks and radial basis function networks both belong to the feedforward family, but each takes a different approach to activation. Radial basis networks use radial basis functions as their activation, placing them in a class of supervised models distinct from the MLP lineage.

  The recurrent neural network stands as the explicit contrast case. Where a feedforward network moves information in one direction, a recurrent network contains loops that allow signals from later stages to influence earlier ones. Feedforward multiplication is essential for backpropagation to work at all. When outputs feed back to the very same inputs and modify them, the result is an infinite loop that backpropagation cannot differentiate through. That is the architectural reason the two families solve different problems.

  The story of the feedforward network is also a story about credit and timing. Legendre and Gauss used a single-weight-layer network with linear activation trained by least squares, which is the bare minimum of the feedforward concept. From that starting point in the early 1800s, the field accumulated the perceptron, multilayer training, stochastic gradient descent, the modern backpropagation formula, and deep learning with gates, often in isolation, often with the work arriving in the literature years or decades after it was done. Ivakhnenko and Lapa's 1965 deep learning algorithm, for instance, preceded the 1986 popularization of backpropagation by more than twenty years.