Skip to content
— CH. 1 · THE BEDROOM TRAINING —

AlexNet

~3 min read · Ch. 1 of 6
6 sections

  • In 2012, Alex Krizhevsky trained a massive neural network inside his parents' bedroom. He used two Nvidia GTX 580 graphics cards sitting on a desk in that small room. Each card held 3GB of video memory and cost US$500 when released. The team needed this specific hardware because the model contained 60 million parameters. A single GPU could not hold all those numbers at once. They split the eight-layer architecture into two halves to fit across both devices. This physical constraint forced a unique design choice for the entire project.

  • The network structure consisted of five convolutional layers followed by three fully connected layers. Some convolutional sections included max-pooling operations to reduce data size. Other parts like layers 3, 4, and 5 connected directly without pooling or normalization steps. The system used ReLU activation functions instead of older tanh or sigmoid types. These non-saturating functions allowed the model to train much faster than previous attempts. Local response normalization appeared between certain layers to improve generalization. Dropout regularization with a drop probability of 0.5 prevented overfitting during the learning process. Biases in specific layers were initialized to constant 1 values to avoid dying neurons.

  • On the 30th of September 2012, the SuperVision team submitted their entry to the ImageNet Large Scale Visual Recognition Challenge. Their final system combined seven different AlexNet models into an ensemble. Five standard versions ran on the ILSVRC-2012 training set containing 1.2 million images. Two variant versions added an extra convolutional layer and trained on 15 million images from Fall 2011. This massive collection achieved a top-5 error rate of 15.3 percent. That result beat the runner-up by more than 10.8 percentage points. Yann LeCun later called this victory an unequivocal turning point in computer vision history.

  • Kunihiko Fukushima proposed the neocognitron concept back in 1980 as an early form of CNN. Yann LeCun developed LeNet-5 in 1989 using supervised learning with backpropagation algorithms. Max pooling appeared in speech processing during 1990 and image processing via Cresceptron in 1992. Researchers like K. Chellapilla adapted these concepts for general-purpose computing in the 2000s. A deep belief network by Raina et al reached 100 million parameters on Nvidia GeForce GTX 280 chips in 2009. Dan Cireșan's team at IDSIA won four competitions between the 15th of May 2011 and the 10th of September 2012. These earlier efforts laid the groundwork but lacked the scale AlexNet eventually achieved.

  • Fei-Fei Li began creating the ImageNet dataset in 2007 to advance visual recognition through large-scale data. The final collection held over 14 million labeled images across 22,000 categories. Amazon Mechanical Turk workers performed the labeling tasks while WordNet organized the hierarchy. Geoffrey Hinton pushed colleagues to adopt neural networks after seeing PASCAL Visual Object Classes results. He found that dataset too small so Malik recommended ImageNet instead. GPU programming advances through Nvidia CUDA platform enabled practical training of these large models. The convergence of big datasets, fast hardware, and better algorithms made success possible. Krizhevsky performed hyperparameter optimization throughout 2012 until the win occurred later that year.

  • As of early 2025, the original paper has been cited over 184,000 times according to Google Scholar. Subsequent research aimed to train increasingly deep CNNs achieving higher performance on benchmarks. GoogLeNet arrived in 2014 followed by VGGNet and Highway network in 2015. ResNet emerged in 2015 as another major milestone in the field. Other teams focused on reproducing AlexNet performance at lower costs with SqueezeNet in 2016. MobileNet appeared in 2017 and EfficientNet in 2019 for mobile devices. DNNResearch formed by the creators sold their company and source code to Google shortly after winning. The Computer History Museum released the original 2012 version under BSD-2 license for public use.

Common questions

Who trained the AlexNet neural network in 2012?

Alex Krizhevsky trained the AlexNet neural network inside his parents' bedroom during 2012. He utilized two Nvidia GTX 580 graphics cards to handle the model's 60 million parameters.

What hardware specifications did the original AlexNet system use?

The AlexNet system used two Nvidia GTX 580 graphics cards each holding 3GB of video memory and costing US$500 when released. The team split the eight-layer architecture across both devices because a single GPU could not hold all 60 million parameters at once.

When did the SuperVision team submit their entry to the ImageNet Large Scale Visual Recognition Challenge?

On the 30th of September 2012, the SuperVision team submitted their entry to the ImageNet Large Scale Visual Recognition Challenge. Their final system combined seven different AlexNet models into an ensemble that achieved a top-5 error rate of 15.3 percent.

How many citations has the original AlexNet paper received as of early 2025?

As of early 2025, the original AlexNet paper has been cited over 184,000 times according to Google Scholar. Subsequent research aimed to train increasingly deep CNNs achieving higher performance on benchmarks following this milestone.

Which earlier neural network concepts influenced the development of AlexNet?

Kunihiko Fukushima proposed the neocognitron concept back in 1980 as an early form of convolutional neural networks. Yann LeCun developed LeNet-5 in 1989 using supervised learning with backpropagation algorithms and Max pooling appeared in speech processing during 1990.

See all questions about AlexNet →

All sources

25 references cited across the entry

  1. 1journalImageNet classification with deep convolutional neural networksAlex Krizhevsky et al. — 2017-05-24
  2. 2webImageNet Large Scale Visual Recognition Competition 2012 (ILSVRC2012)
  3. 3webNVIDIA GeForce GTX 580 Specs2024-11-12
  4. 4webcalflops: a FLOPs and Params calculate tool for neural networks
  5. 5inlineSuperVision team invited talk at the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) workshop on 2012-10-12, held within the ECCV 2012, in Florence, Italy.
  6. 6journalNeocognitronK. Fukushima — 2007
  7. 7journalNeocognitron: A Self-organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in PositionKunihiko Fukushima — 1980
  8. 8journal <!-- citation bot bypass-->Backpropagation Applied to Handwritten Zip Code RecognitionY. LeCun — MIT Press - Journals — 1989
  9. 9journalGradient-based learning applied to document recognitionYann LeCun — 1998
  10. 10conferenceA Neural Network for Speaker-Independent Isolated Word RecognitionKouichi Yamaguchi et al. — November 1990
  11. 11conferenceCresceptron: a self-organizing neural network which grows adaptivelyJ. Weng — IEEE — 1992
  12. 12bookTenth International Workshop on Frontiers in Handwriting RecognitionKumar Chellapilla et al. — Suvisoft — 2006
  13. 13conferenceLarge-scale deep unsupervised learning using graphics processorsRajat Raina — ACM — 2009-06-14
  14. 14journalFlexible, High Performance Convolutional Neural Networks for Image ClassificationDan Cireșan — 2011
  15. 15webIJCNN 2011 Competition result table2010
  16. 16webHistory of computer vision contests won by deep CNNs on GPUJürgen Schmidhuber — 17 March 2017
  17. 17book2012 IEEE Conference on Computer Vision and Pattern RecognitionDan Cireșan et al. — Institute of Electrical and Electronics Engineers (IEEE) — June 2012
  18. 18journalMax-Margin Markov NetworksBen Taskar et al. — MIT Press — 2003
  19. 19bookDive into deep learningAston Zhang et al. — Cambridge University Press — 2024
  20. 20bookThe worlds I see: curiosity, exploration, and discovery at the dawn of AIFei Fei Li — Moment of Lift Books ; Flatiron Books — 2023
  21. 21webCHM Releases AlexNet Source Codehhackford — 2025-03-20
  22. 22webHow a stubborn computer scientist accidentally launched the deep learning boom11 November 2024
  23. 23inlineAlexNet paper on Google Scholar
  24. 24webcuda-convnet: High-performance C++/CUDA implementation of convolutional neural networksAlex Krizhevsky — July 18, 2014
  25. 25citationcomputerhistory/AlexNet-Source-CodeComputer History Museum — 2025-03-22