Attention (machine learning)

• 1. Introduction

  ›

  Attention, in machine learning, is a method that determines how much weight each part of a sequence deserves relative to every other part. It expresses that importance through numbers called soft weights. When a trained network translates the phrase 'I love you' into French, those weights become visible. On the first pass of the decoder, 94% of the attention weight falls on the word 'I', and the network produces 'je'. On the second pass, 88% of the weight moves to 'you', yielding 't''. On the final pass, 95% concentrates on 'love', completing 'aime'. The model arrived at correct French word order without any explicit instruction about French syntax. It found the right source word at each step by attending to it directly. How did that idea develop from mid-century psychology into the mechanism behind models like BERT, T5, and GPT? And what made a 2017 paper titled 'Attention is All You Need' the pivot point for a field that had relied on recurrence for decades?

  43912

• 2. The Recurrence Problem

  ›

  Recurrent neural networks process language one word at a time, passing context forward through a hidden state vector. Each new word updates that state, but earlier words grow harder to recall as the sequence lengthens. The network inherently favored information near the end of a sentence, treating recent words as more significant and attenuating earlier ones. In a long sentence, the first word could contribute almost nothing to the final representation.

  The cocktail party effect, studied in psychology and biology research from the 1950s-1960s, described how humans focus on relevant signals while filtering noise. That observation pointed toward a computational version of selective attention. Sigma-pi units appeared in the 1980s. In 1992, fast weight controllers had one neural network produce the weights of another through outer products; that technique was later renamed linearized self-attention. The bilateral filter, introduced in image processing in 1998, showed that pairwise affinity matrices could propagate relevance across elements of a spatial structure. Non-local means, developed in 2005, extended that affinity-based approach to image denoising using Gaussian similarity kernels as fixed, attention-like weights.

  The direct solution for language translation arrived in 2014, when researchers grafted attention onto an encoder-decoder architecture, with particular gains on long sentences. Attention gave each output token a direct path to any input token. The sequential hidden-state bottleneck was bypassed entirely.

  By 2016, self-attention had been integrated into RNN-based models to capture dependencies within a single sequence. That step proved transitional: within a year, a paper would argue that recurrence could be removed from the architecture altogether.

  42917401

• 3. Attention Is All You Need

  ›

  'Attention is All You Need', published in 2017, introduced the Transformer: an architecture that removed recurrence and built its processing entirely around attention. Self-attention was the core operation: every element of the input sequence attends to every other element simultaneously. That parallel structure replaced the sequential RNN with a mechanism that could run across many hardware cores at once. Training became substantially faster.

  Self-attention computes three vectors for each input position: a query, a key, and a value. Each position's query is compared against all positions' keys to produce a relevance score. Those scores determine how much of each position's value to include in the output. When all three vectors come from the same model, the result is self-attention. Every position has equal access to every other position's information in a single pass.

  BERT, T5, and the generative pre-trained transformers known as GPT all emerged from the Transformer design. Each applied the same core mechanism to different tasks. Relation networks and set Transformers, developed in the same year as the original paper, applied attention to unordered sets and relational reasoning. The mechanism had spread beyond language before 2017 was out.

  3921224445

• 4. Shapes of Attention

  ›

  Bahdanau-style attention, also called additive attention, uses a learned weight matrix to combine encoder and decoder hidden states before computing an alignment score. It handled the task of matching source words to target words by treating alignment as a learnable function. Luong-style attention takes a multiplicative approach: a dot product between encoder and decoder states, scaled by a single learnable matrix. The two styles reflect different answers to the question of how to measure relevance between sequence elements.

  Multi-head attention runs several attention operations in parallel, each with its own set of learned projections for queries, keys, and values. The outputs are concatenated and projected through another learned matrix. Running multiple heads lets the model capture different relationship types within the same sequence simultaneously.

  For convolutional neural networks, spatial attention focuses on which regions of an input image contribute most to a prediction. Channel attention operates on a different axis: which feature maps deserve the most weight. The two can be combined so that a model is selective along both dimensions at once.

  Masked attention imposes a hard constraint on autoregressive decoding. The mask is an upper-triangular matrix: weights above the diagonal are forced to zero. Each output position can only attend to prior outputs, not to positions it has yet to generate.

  Flash attention addresses a fundamental scaling problem: the attention matrix grows proportional to the square of the number of input tokens. As sequences grow longer, that quadratic relationship consumes large amounts of GPU memory. Flash attention partitions the computation into blocks that fit in the chip's faster on-chip memory. This sidesteps the need to store large intermediate matrices while achieving the same mathematical result. Meta developed a related kernel called FlexAttention. It lets users modify attention scores before the softmax step, and dynamically selects the optimal algorithm for each configuration.

  Non-local neural networks carried attention principles into computer vision in 2018, capturing long-range dependencies across spatial positions and time. Graph attention networks, also developed in 2018, applied the mechanism to data organized in irregular graph structures rather than fixed sequences.

  4202456

• 5. A Field Transformed

  ›

  AlphaFold applied Transformer attention to protein folding, carrying the architecture into structural biology. CLIP used it for vision-language pretraining, combining visual and textual processing in a single model. Dense segmentation models CCNet and DANet extended attention to pixel-level prediction across full images.

  Vision transformers, abbreviated ViTs, achieved competitive results in image classification from 2019 onward. Question answering and text summarization in natural language processing both benefited from attention's capacity to hold global context across long inputs. Speech recognition also joined the list of tasks where attention improved performance.

  Reformer, Linformer, and Performer, three efficient Transformer variants developed between 2019 and 2020, introduced scalable approximations of attention for long sequences. Each traded some mathematical precision for the ability to handle inputs that standard attention designs could not process economically. As these variants spread, researchers turned to a pressing question: does high attention weight reveal why a model made its decision?

  5354555652

• 6. Reading the Maps

  ›

  Saliency maps, also called attention maps, became a standard inspection tool after the original vision transformer paper. To produce one, a researcher computes attention scores for a specific head and layer, then renders them as a heat map. Deeper layers of the network tend to yield more semantically meaningful visualizations: later-layer maps often highlight recognizable structures rather than low-level patterns.

  Attention rollout combines scores from all layers into a single view. The algorithm is recursive. It computes the dot product of successive attention maps layer by layer, tracing how scores propagate from input to output across the full network.

  Because vision transformers are typically trained in a self-supervised manner, their attention maps are generally not class-sensitive: they show what the model weighted, not which category it assigned. Class-discriminative attention maps, abbreviated CDAM, address that gap by combining attention scores with gradients computed for the class token. That gradient information focuses the visualization on regions relevant to the specific category the model predicted. GradCAM, originally developed for convolutional networks, achieves a similar result. It back-propagates gradients to the outputs of the final attention layer and can be applied to vision transformers as well.

  Higher attention scores do not always correlate with greater impact on model performance, and this disconnect has generated ongoing debate. A position can receive high attention weight and yet have little measurable effect on the final output. Hopfield networks, originally conceived as associative memory systems, were reinterpreted after 2019 as attention mechanisms. Surveys by Niu et al. and Soydaner document this evolving picture: the boundary between associative memory and attention may be less fixed than the 2014 formulation implied.

  7891051