GloVe

• 1. Introduction

  ›

  GloVe, which stands for Global Vectors, is an algorithm that teaches computers to understand words by turning them into numbers. Specifically, into points in space. The premise sounds abstract until you see what it actually does: two words with similar meanings end up geometrically close together. Distance becomes a measure of meaning. That idea, simple to state and tricky to build, drove researchers at Stanford University to launch GloVe in 2014 as an open-source project and a direct competitor to an already-popular system called word2vec. The questions worth sitting with are: how do you capture the meaning of a word in a set of numbers? What does it mean for two words to be statistically similar? And why, after all the effort, have both GloVe and word2vec been superseded by something else entirely?

• 2. A Quote and a Co-occurrence

  ›

  The linguist J. R. Firth wrote in 1957 that "you shall know a word by the company it keeps." GloVe takes that observation literally. Its central mechanism is counting how often two words appear near each other across an entire body of text. When the word "model" appears at position eight in a sentence and the context window is set to three, the algorithm checks whether words within three positions in either direction qualify as co-occurring. A word is not counted as occurring in the context of itself, though it counts again if it appears a second time within that same window. The result of sweeping across a full corpus is a large table of co-occurrence counts, one entry for every pair of words in the vocabulary.

• 3. What the Numbers Reveal

  ›

  In a corpus of six billion tokens, the ratio of co-occurrence probabilities becomes a surprisingly clean signal. Consider the words "ice" and "steam." Both co-occur frequently with "water" and rarely with "fashion", so those words do not help distinguish the two. But "solid" co-occurs far more often with "ice" than with "steam", and "gas" co-occurs far more often with "steam" than with "ice." By looking at the ratio of those co-occurrence probabilities rather than the raw counts, GloVe captures meaningful contrasts that raw frequency alone would obscure. The key insight is that words with similar co-occurrence probability profiles should receive similar vector representations in the final model.

• 4. The Learning Mechanism

  ›

  GloVe sits at the intersection of two approaches that had previously been treated as separate. Global matrix factorization methods compress a large co-occurrence matrix into dense vectors; local context window methods train on one small passage at a time. GloVe combines both by framing the training task as a log-bilinear regression: for each word, two vectors are learned simultaneously, and the goal is to make their dot product predict the logarithm of the co-occurrence count. To prevent rare word pairs from contributing excessive noise, the squared error for each pair is scaled by a weighting function that ramps up gradually as the raw count rises. The authors of the original paper reported that a specific set of hyperparameters performed well in practice, and those values became the reference defaults.

• 5. From Training to Representation

  ›

  Each word in the vocabulary ends up with four trained parameters once training is finished. Two of those parameters are the context-specific terms that become irrelevant after training is complete. What remains are two vectors, and the authors recommended adding them together as the final representation, because doing so empirically outperformed using either one alone. With those vectors in hand, the model can do things like identify synonyms, map company names to their products, and connect zip codes to the cities they belong to. The SpaCy natural language processing library adopted GloVe to build its semantic word embedding features, using distance measures such as cosine similarity and Euclidean distance to rank how closely words match.

• 6. Limits and a Different Problem

  ›

  GloVe cannot handle homographs, words that share a spelling but carry different meanings. The word "bank" beside a river and "bank" as a financial institution look identical to GloVe, which produces a single vector for any given spelling regardless of context. That single-vector limitation is a direct consequence of the unsupervised co-occurrence approach: every occurrence of a word, whatever its sense, feeds into the same representation. One domain where researchers pressed past that limit was mental health: GloVe was used as the word representation layer in both online and offline systems designed to detect psychological distress in patient interviews, a task that demanded nuanced semantic distinctions from a model that, by design, collapses all senses into one.

• 7. The Transformer Era

  ›

  BERT is the name most associated with what replaced both GloVe and word2vec. Transformer-based models like BERT add multiple neural-network attention layers on top of a word embedding approach similar to word2vec, allowing them to assign different representations to the same word depending on surrounding context. That flexibility addresses the homograph problem directly and, over time, these approaches came to be regarded as the state of the art in natural language processing. Both GloVe and word2vec are now considered outdated in the field, though the core intuition that GloVe crystallized in 2014 from J. R. Firth's 1957 observation remains embedded in the foundations of every model that followed.