Unsupervised learning

• 1. Introduction

  ›

  Unsupervised learning is a framework in machine learning where algorithms find patterns in data that nobody has labeled. No teacher, no classifications, no hand-tagged examples. The machine is handed raw material and told, in essence, to figure it out. How does a computer learn anything useful without being told what it is looking at? And what kinds of machines have researchers built to make that possible? Those questions drive everything that follows.

• 2. The Data Advantage

  ›

  Common Crawl, the massive repository of web text built by crawling the internet with only minor filtering, exemplifies what makes unsupervised learning so practical. Harvesting data "in the wild" at enormous scale costs comparatively little. Supervised learning, by contrast, depends on manually constructed datasets. The ImageNet1000 benchmark, for instance, required careful human effort to build and label. That expense is the practical ceiling on supervised systems. Unlabeled data is everywhere and cheap, which makes unsupervised methods attractive at scale. The generative pretraining method illustrates the payoff: a model first trains to generate a large text corpus, then gets fine-tuned for a specific task like text classification, extracting far more value from the original unlabeled material.

• 3. Tasks and the Hazy Line

  ›

  Tasks in machine learning tend to split into two broad camps: discriminative tasks, which involve recognition, and generative tasks, which involve imagination. Unsupervised methods lean toward the generative side, but the boundary is less crisp than it first appears. Image recognition is the clearest example of how fluid this can be. It started as a heavily supervised problem, shifted to a hybrid approach that incorporated unsupervised pre-training, and then swung back toward supervision again when techniques like dropout and ReLU proved effective. A typical generative task works by removing part of a data point and asking the model to reconstruct the missing piece. Denoising autoencoders and BERT both follow that logic, learning by filling in what has been taken away.

• 4. A Family of Networks

  ›

  John Hopfield described a class of neural network in 1982 that functions as a content-addressable memory and classifier, drawing on an Ising magnetic model first proposed for cognition in 1974. Hopfield networks use deterministic binary neurons with symmetric connections and no self-connections. Geoff Hinton and Terry Sejnowski extended that work a year later, introducing the Boltzmann machine with probabilistic neurons and following earlier work by Sherington and Kirkpatrick from 1975. Paul Smolensky published Harmony Theory in 1986, which described what was effectively a Restricted Boltzmann Machine with the same energy function, though Smolensky did not provide a practical training scheme. Hinton supplied that in the mid-2000s. Later, in 1995, Peter Dayan and Hinton introduced the Helmholtz machine, and Hochreiter and Schmidhuber introduced the LSTM neuron that same year for language tasks. The names on some of these networks are telling: Boltzmann and Helmholtz were not themselves artificial intelligence researchers. Their contributions came from physics and physiology, but the analytical methods they developed proved directly applicable. Of all the people whose names appear on these architectures, only Hopfield worked directly with neural networks.

• 5. Energy, Harmony, and Learning Rules

  ›

  Ludwig Boltzmann's analysis of a gas gives the energy function its name and its intuition. Boltzmann described how the macroscopic energy of a gas arises from the microscopic probabilities of particle motion, using the Boltzmann constant and temperature as anchors. Neural researchers borrowed that framework and applied it to networks. In a Boltzmann machine, energy plays the role of the cost function, and the network seeks configurations of low energy, which Paul Smolensky called high Harmony. Training unsupervised networks draws on methods that differ substantially from the backpropagation dominant in supervised settings. The toolkit includes the Hopfield learning rule, the Boltzmann learning rule, Contrastive Divergence, Wake-Sleep, Variational Inference, Maximum Likelihood, Maximum A Posteriori, and Gibbs Sampling. One practical limitation stands out across several of these approaches: the Boltzmann machine's freedom of connections makes it difficult to analyze, and reaching equilibrium requires too many iterations. The Restricted Boltzmann Machine addressed both problems by removing lateral connections within a layer, which made the training scheme faster and more practical.

• 6. Hebbian Roots and Self-Organization

  ›

  Donald Hebb's principle, that neurons that fire together wire together, is the classical example of unsupervised learning in the study of neural networks. Hebbian learning reinforces a connection between two neurons based solely on coincident activity, not on error. A related extension called spike-timing-dependent plasticity, or STDP, modifies synaptic weights based on the time interval between action potentials. Researchers have proposed Hebbian learning as an explanation for pattern recognition and experiential learning in biological brains. Two architectures built on related principles remain widely used. The self-organizing map produces a topographic organization where nearby positions in the map correspond to inputs with similar properties. The adaptive resonance theory model, known as ART, lets cluster counts vary with problem size and gives the user a parameter called the vigilance parameter to control how similar members of the same cluster must be. ART networks are applied to tasks including automatic target recognition and seismic signal processing.

• 7. Probabilistic Methods and Latent Variables

  ›

  Topic modeling is one of the most concrete illustrations of what latent variable models do. A document contains observable words, but the topic that generated those words is hidden, a latent variable that shifts the statistical parameters underlying word choice. The method of moments connects unknown model parameters to the moments of random variables, which can then be estimated from data. First-order moments are mean vectors; second-order moments are covariance matrices when the mean is zero; higher-order moments are represented as tensors. Tensor decomposition techniques have been shown to consistently recover the parameters of a large class of latent variable models under certain assumptions. The Expectation-Maximization algorithm, or EM, is a widely used alternative for the same class of problems. Its practical drawback is a tendency to get stuck in local optima rather than converging to the true parameters. The method of moments offers a theoretical advantage there: global convergence is guaranteed under certain conditions, which makes it a stronger tool when that guarantee matters. Kingma, Rezende, and colleagues introduced Variational Autoencoders in 2013 as a Bayesian graphical probability network with neural nets as components, and VAEs now serve as one of the primary tools for generating realistic data from learned distributions.