ImageNet
In 2006, computer scientist Fei-Fei Li began working on a project to build a massive visual database. She wanted to expand the data available for training artificial intelligence algorithms at a time when most research focused only on models and code. By 2007, she had met with Princeton professor Christiane Fellbaum to discuss using WordNet as a foundation. The team started labeling images in July 2008 and finished in April 2010. They used Amazon Mechanical Turk to help classify the pictures. A total of 49,000 workers from 167 countries filtered and labeled over 160 million candidate images. Each of the final 14 million images received three separate labels. The original plan called for 10,000 images per category across 40,000 categories. That would have required 400 million images verified three times each. Human workers could classify at most two images per second. At that speed, the task would take 19 human years of labor without rest. The team presented their database for the first time as a poster at the 2009 Conference on Computer Vision and Pattern Recognition in Florida.
On the 30th of September 2012, a convolutional neural network named AlexNet achieved a top-5 error rate of 15.3 percent in the ImageNet challenge. This result was more than 10.8 percentage points lower than the runner-up. Graphics processing units made training these networks feasible during this period. According to The Economist, people started paying attention not just within the AI community but across the technology industry as a whole. In 2015, Microsoft's very deep CNN with over 100 layers won the contest with an error rate of 3.57 percent. Andrej Karpathy estimated in 2014 that a human could reach an error rate of 2.4 percent with maximal effort. By 2017, 29 out of 38 competing teams had greater than 95 percent accuracy. The dramatic quantitative improvement marked the start of an industry-wide artificial intelligence boom. The winning entry for object localization in 2013 was OverFeat, an architecture for simultaneous classification and localization. The winning entry for classification in 2016 was CUImage, an ensemble model combining six different networks.
The full original dataset is referred to as ImageNet-21K and contains 14,197,122 images divided into 21,841 classes. Some papers round this up and name it ImageNet-22K. The full set was released in Fall of 2011 as fall11_whole.tar. There is no official train-validation-test split for ImageNet-21K. Some classes contain only one to ten samples while others contain thousands. Each synset in WordNet 3.0 has a unique identifier starting with n because ImageNet includes only nouns. For example, the wnid of synset dog is n02084071. The categories fall into nine levels from level one such as mammal to level nine such as German shepherd. In 2012, ImageNet was the world's largest academic user of Mechanical Turk. The average worker identified fifty images per minute. ImageNet consists of images in RGB format with varying resolutions ranging from 4288 by 2848 down to 75 by 56 pixels. Dense SIFT features were available for download designed for bag of visual words.
The resulting annual competition became known as the ImageNet Large Scale Visual Recognition Challenge or ILSVRC. It uses a trimmed list of only 1000 image categories including ninety of the 120 dog breeds classified by the full schema. The first competition in 2010 had eleven participating teams and achieved 52.9 percent classification accuracy. The second competition in 2011 saw another SVM win at top-5 error rate 25 percent. By 2014 more than 50 institutions participated in the challenge. The winning entry for localization in 2017 was VGGNet while the classification winner was GoogLeNet. In 2017, 29 out of 38 competing teams had greater than 95 percent accuracy. The organizers stated that the 2017 competition would be the last one since the benchmark had been solved. They also announced plans to organize a new competition on three-dimensional images. However, such a competition never materialized. The dataset is expected to be smaller because creating 3D data costs more than annotating pre-existing two-dimensional images.
During the 2018 to 2020 period, they removed the download of ImageNet-21K as they went through extensive filtering in person synsets. Out of 2832 synsets, 1593 were deemed potentially offensive. Of the remaining 1239, 1081 were deemed not really visual. Only 158 synsets remained from this group. In 2021 winter, 2702 categories in the person subtree were removed to prevent problematic behaviors in trained models. Only 130 synsets in the person subtree remained after these changes. In 2021, ImageNet-1K was updated by blurring faces appearing in 997 non-person categories. They found that 243,198 images contain at least one face out of all 1,431,093 images. The total number of faces adds up to 562,626. Training models on the dataset with blurred faces caused minimal loss in performance. It is estimated that over 6 percent of labels in the validation set are wrong. A study described how bias is deeply embedded in most classification approaches for all sorts of images.
Up Next
Common questions
When did Fei-Fei Li start working on the ImageNet project?
Computer scientist Fei-Fei Li began working on a project to build a massive visual database in 2006. The team started labeling images in July 2008 and finished in April 2010.
What was the error rate of AlexNet during the ImageNet challenge on the 30th of September 2012?
A convolutional neural network named AlexNet achieved a top-5 error rate of 15.3 percent in the ImageNet challenge on the 30th of September 2012. This result was more than 10.8 percentage points lower than the runner-up.
How many images are contained in the full original ImageNet dataset known as ImageNet-21K?
The full original dataset referred to as ImageNet-21K contains 14,197,122 images divided into 21,841 classes. The full set was released in Fall of 2011 as fall11_whole.tar.
Why did the organizers stop holding the ImageNet Large Scale Visual Recognition Challenge after 2017?
The organizers stated that the 2017 competition would be the last one since the benchmark had been solved. They also announced plans to organize a new competition on three-dimensional images but such a competition never materialized.
What changes were made to the ImageNet dataset regarding person categories during 2021?
In 2021 winter, 2702 categories in the person subtree were removed to prevent problematic behaviors in trained models. Only 130 synsets in the person subtree remained after these changes.
All sources
62 references cited across the entry
- 1newsNew computer vision challenge wants to teach robots to see in 3D7 April 2017
- 2newsFor Web Images, Creating New Technology to Seek and FindJohn Markoff — 19 November 2012
- 3webImageNet2020-09-07
- 4newsFrom not working to neural networking25 June 2016
- 5webImageNet OverviewImageNet
- 6magazineFei-Fei Li's Quest to Make AI Better for HumanityJesse Hempel — 13 November 2018
- 7journalRecognition-by-components: A theory of human image understanding.Irving Biederman — 1987
- 8webHow a stubborn computer scientist accidentally launched the deep learning boomTimothy B. Lee — 2024-11-11
- 9conferenceWhere have we been? Where are we going?Fei-Fei Li et al. — 2017
- 10webTowards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet HierarchyKaiyu Yang et al. — 2019-09-17
- 13webThe data that transformed AI research—and possibly the worldDave Gershgorn — Atlantic Media Co. — 26 July 2017
- 14citation2009 conference on Computer Vision and Pattern RecognitionJia Deng et al. — 2009
- 15citationHow we're teaching computers to understand picturesFei-Fei Li — 23 March 2015
- 17journalImageNet classification with deep convolutional neural networksAlex Krizhevsky et al. — June 2017
- 18newsMachines 'beat humans' for a growing number of tasks30 November 2017
- 19webThe inside story of how AI got good enough to dominate Silicon ValleyDave Gershgorn — 18 June 2018
- 20book2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)Kaiming He et al. — 2016
- 22webWhat I learned from competing against a ConvNet on ImageNetAndrej Karpathy — 2014-09-02
- 23journalImageNet Large Scale Visual Recognition ChallengeOlga Russakovsky et al. — 2015-12-01
- 24webImageNet Summary and Statistics (updated on April 30, 2010)2013-01-15
- 25webImageNet API documentation2013-01-22
- 28webImageNet2013-04-05
- 30webImageNet
- 31webImageNet
- 32bookTrends and Topics in Computer VisionOlga Russakovsky et al. — Springer — 2012
- 33arxivImageNet-21K Pretraining for the MassesTal Ridnik et al. — 2021-08-05
- 34webImageNet
- 35bookProceedings of the 2020 Conference on Fairness, Accountability, and TransparencyKaiyu Yang et al. — ACM — 2020-01-27
- 38journalA Study of Face Obfuscation in ImageNetKaiyu Yang et al. — PMLR — 2022-06-28
- 39arxivBenchmarking Neural Network Robustness to Common Corruptions and PerturbationsDan Hendrycks et al. — 2019
- 40journalDo ImageNet Classifiers Generalize to ImageNet?Benjamin Recht et al. — PMLR — 2019-05-24
- 42bookCVPR 2011Yuanqing Lin et al. — IEEE — June 2011
- 43bookCVPR 2011Jorge Sanchez et al. — IEEE — June 2011
- 44bookComputer Vision – ECCV 2010Florent Perronnin et al. — Springer — 2010
- 47journalDetecting Avocados to Zucchinis: What Have We Done, and Where Are We Going?Olga Russakovsky et al. — 2013
- 48arxivOverFeat: Integrated Recognition, Localization and Detection using Convolutional NetworksPierre Sermanet et al. — 2013
- 49book2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)Christian Szegedy et al. — IEEE — June 2015
- 50newsThe Quartz guide to artificial intelligence: What is it, why is it important, and should we be afraid?Dave Gershgorn — 10 September 2017
- 51newsA Learning Advance in Artificial Intelligence Rivals Human AbilitiesJohn Markoff — 10 December 2015
- 52newsForget the Turing test – there are better ways of judging AIJacob Aron — 21 September 2015
- 53webIlsvrc2016
- 54conferenceAggregated Residual Transformations for Deep Neural NetworksSaining Xie et al. — 2017
- 55arxivSqueeze-and-Excitation NetworksJie Hu et al. — 2017
- 56citationPervasive Label Errors in Test Sets Destabilize Machine Learning BenchmarksCurtis G. Northcutt et al. — 2021-11-07
- 57citationAre we done with ImageNet?Lucas Beyer et al. — 2020-06-12
- 59newsThe viral selfie app ImageNet Roulette seemed fun – until it called me a racist slurJulia Carrie Wong — 18 September 2019
- 60webExcavating AI: The Politics of Training Sets for Machine LearningKate Crawford et al. — 19 September 2019
- 61arxivExcavating "Excavating AI": The Elephant in the GalleryMichael J. Lyons — 2020