Convolutional neural network

• 1. Introduction

  ›

  Convolutional neural networks power the face-unlock on your phone, flag tumors in mammograms, and helped a machine beat the world's best Go players. Yet the core idea behind them traces back to experiments on cats in the 1950s and 1960s, when researchers David Hubel and Wiesel studied how individual neurons in a cat's visual cortex responded to nothing more than light falling on a tiny patch of the animal's visual field. That biological observation, painstakingly replicated in laboratories over decades, became the blueprint for an entire class of artificial networks that now underpin computer vision. How did a curiosity about cat eyes become the de-facto standard architecture for image recognition? And what exactly happens inside one of these networks when it looks at a photograph?

• 2. Roots in the Visual Cortex

  ›

  Hubel and Wiesel's 1968 paper identified two distinct types of neurons in the visual brain. Simple cells gave their strongest response to straight edges oriented in a specific direction inside their receptive field. Complex cells had larger receptive fields and were indifferent to the exact position of an edge. The researchers also proposed a cascading model in which these two cell types could work together for pattern recognition.

  Kunihiko Fukushima took that cascading idea and built a computational counterpart. In 1969 he introduced a multilayer visual feature-detection network in which every element within a given layer shared the same interconnecting coefficients. That shared-weight arrangement is the essential core of what we now call a convolutional network, though in this early version the weights were not trained by any algorithm. In the same paper, Fukushima also introduced the ReLU activation function, a mathematical operation that would later become standard across the entire field.

  Fukushima went further in 1980 with the neocognitron. It formalized two types of layers that persist in modern architectures: an S-layer, which is a shared-weights receptive-field layer later known as a convolutional layer, and a C-layer, a downsampling layer whose units compute a weighted average of activations in a patch. The downsampling and competitive inhibition in the C-layer allowed the network to recognize features even when objects shifted position in the image. Separate supervised and unsupervised learning algorithms were proposed over the following decades to train the neocognitron's weights, though today CNN architectures are almost universally trained through backpropagation.

• 3. The Trainable Network Arrives

  ›

  The term convolution first appeared in a neural-network paper at the Conference on Neural Information Processing Systems in 1987, written by Toshiteru Homma, Les Atlas, and Robert Marks II. They replaced multiplication with convolution in the time domain, providing shift invariance and linking the approach to signal-processing concepts. Their demonstration ran on a speech recognition task.

  Also in 1987, Alex Waibel and colleagues introduced the time delay neural network, or TDNN, for phoneme recognition. It was the first convolutional network to combine weight sharing with training by gradient descent via backpropagation. TDNNs processed speech signals time-invariantly, and a 1990 variant by Hampshire and Waibel added a second dimension of convolution so the system became invariant to both time and frequency shifts.

  Yann LeCun and colleagues in 1989 applied backpropagation to learn convolution kernel coefficients directly from images of hand-written numbers, making learning fully automatic. That same year, Denker and colleagues had designed a two-dimensional CNN to recognize hand-written ZIP codes, but had to hand-design all the kernel coefficients for lack of an efficient training method. LeCun's automatic approach outperformed manual coefficient design and proved suited to a broader range of image types.

  Wei Zhang and colleagues had independently used backpropagation to train convolution kernels for alphabet recognition in 1988, calling the model a shift-invariant pattern recognition neural network before the CNN name was coined in the early 1990s. They later removed the final fully connected layer and applied the same architecture to medical image segmentation in 1991 and to breast cancer detection in mammograms in 1994.

• 4. LeNet and the GPU Era

  ›

  LeNet-5, a seven-level convolutional network published by LeCun and colleagues in 1995, classified hand-written numbers on checks digitized in 32x32-pixel images. It outperformed other commercial check-reading systems available at the time. NCR integrated it into check-reading systems deployed at several American banks starting in June 1996, where it read millions of checks per day.

  For roughly two decades after that, CNNs remained computationally expensive. The breakthrough in the 2000s required fast implementations on graphics processing units. In 2004, K. S. Oh and K. Jung showed that standard neural networks could run about 20 times faster on a GPU than on a CPU. The first GPU implementation of a CNN specifically was described in 2006 by K. Chellapilla and colleagues, who achieved a four-times speedup over an equivalent CPU implementation.

  Dan Ciresan and colleagues at IDSIA trained deep feedforward networks on GPUs in 2010, and extended the approach to CNNs in 2011, reporting a 60-times speedup over CPU training. That same year their network won an image recognition contest, achieving superhuman performance for the first time. AlexNet, a GPU-based CNN by Alex Krizhevsky and colleagues, then won the ImageNet Large Scale Visual Recognition Challenge in 2012 and has been described as an early catalytic event for the broader AI boom.

• 5. Inside the Architecture

  ›

  A CNN processes data as a tensor with four dimensions: number of inputs, height, width, and channels. Each convolutional layer slides a small filter, called a kernel, across the input to produce a feature map. A 5x5 kernel, for instance, requires only 25 shared weights, compared to the 10,000 weights that each neuron in a fully connected layer would need to process a 100x100 image. That dramatic compression is what allows CNNs to grow deeper without the vanishing-gradient problems that plagued earlier networks.

  Pooling layers follow convolutional layers and reduce the spatial dimensions of feature maps. The most common form takes the maximum value in each small region, typically a 2x2 window with a stride of 2, which discards 75% of activations. Max pooling was introduced by Yamaguchi and colleagues in 1990 as part of a speaker-independent word recognition system. J. Weng and colleagues introduced it into the vision field in 1993 through a neocognitron variant called the cresceptron.

  The ReLU activation function removes negative values from feature maps by setting them to zero, introducing nonlinearity without distorting the receptive fields of convolution layers. In 2011, Xavier Glorot, Antoine Bordes, and Yoshua Bengio found that ReLU enabled better training of deeper networks compared to activation functions that had been dominant before that year. ReLU trains networks several times faster than those older alternatives without a significant penalty to accuracy.

  Fully connected layers sit at the end of the network and handle final classification, connecting every neuron to every neuron in the adjacent layer. Regularization techniques combat overfitting throughout training: dropout, introduced in 2014, randomly drops individual nodes from the network during each training stage with probability of 0.5, effectively forcing the network to learn more robust features that transfer better to new data.

• 6. Where CNNs Are Applied

  ›

  In 2012, an error rate of 0.23% on the MNIST handwriting database was reported using a CNN. At the ImageNet Large Scale Visual Recognition Challenge in 2014, almost every highly ranked team used a CNN as their core framework. The winning network, GoogLeNet, applied more than 30 layers and raised the mean average precision of object detection to 0.439329 while cutting classification error to 0.06656. Performance at that level was close to human accuracy on the same test set.

  In December 2014, Clark and Storkey published a paper showing that a CNN trained on a database of human professional Go games could outperform the traditional program GNU Go. A subsequent large 12-layer CNN correctly predicted the professional move in 55% of positions, matching the accuracy of a 6-dan human player, and beat GNU Go in 97% of games when used directly to play without any search. AlphaGo, which became the first system to beat the best human Go player, used two CNNs: one as a policy network to choose candidate moves and one as a value network to evaluate board positions.

  In 2015, Atomwise introduced AtomNet, the first deep learning neural network for structure-based drug design. The system learns chemical features such as aromaticity, sp3 carbons, and hydrogen bonding from three-dimensional representations of chemical interactions, in the same way image networks compose local features into larger structures. AtomNet was subsequently used to predict candidate treatments for Ebola and multiple sclerosis.

  From 1999 to 2001, Fogel and Chellapilla published papers showing how a CNN could learn to play checkers through co-evolution, without ever studying prior human games. Their program, Blondie24, was tested on 165 games and ranked in the highest 0.4% of players. It also earned a win against the program Chinook at its expert level of play. The same architecture that reads mammograms and recognizes speech also learned checkers strategy from scratch.