Contrastive Language-Image Pre-training

• 1. Introduction

  ›

  Contrastive Language-Image Pre-training, known as CLIP, can classify a photograph into any category on earth. It does this without having been trained on that specific category at all. The method, developed originally by OpenAI, taught two neural networks to share a common mathematical space. One network learned to read images; the other learned to read text. When both outputs live in that shared space, a photograph and a phrase can be compared directly. That single insight came to underpin tools that now generate, search, and rank images across the internet. The training data that made this possible came from hundreds of millions of internet image-caption pairs. That dataset has never been publicly released.

  1910

• 2. Learning by Contrast

  ›

  The multi-class N-pair loss governs everything CLIP learns during training. The model processes batches of image-caption pairs; for each batch, it computes a vector for every image and a vector for every caption. The loss function works in two directions at once. It encourages high dot products between matching image-text pairs while discouraging high dot products between non-matching ones. This symmetric cross-entropy across all similarity scores in a batch implements the contrastive objective.

  A single temperature parameter, learned during training, controls how sharply the model distinguishes similar pairs from dissimilar ones. SigLIP, a later variant, proposed replacing this loss with a negative log sigmoid function. SigLIP's approach evaluates each pair in isolation rather than measuring it against all others in the batch simultaneously. Whether using the original loss or SigLIP's modification, the result is two encoders whose outputs can be directly compared. The architecture behind each encoder, from pixel-level choices to text vocabulary, shapes what that comparison can resolve.

  561

• 3. Vision and Language

  ›

  Vision transformers, called ViTs, serve as the primary image encoder in CLIP models. A name like ViT-L/14 carries two pieces of information. The letter L marks the model as large within a scale running from base through large, huge, and giant. The number 14 means the image is divided into 14-by-14 pixel patches before the transformer processes it. The image and text encoders must produce vectors of identical length; that length is called the embedding dimension. In OpenAI's original models, embedding dimensions ran from 512 to 1024 for the ResNet series and from 512 to 768 for the ViTs.

  OpenAI's ResNet image encoders departed from the standard architecture in three ways. At the input stem, three stacked 3x3 convolutions replaced the conventional single 7x7. At the start of each downsampling layer, an average pooling step of stride 2 blurred the image before reduction. The final convolutional layer used multiheaded attention pooling at its output. LAION's OpenCLIP series later added ConvNeXt as a third image encoder option.

  The text encoder in the original OpenAI CLIP model was a 63-million-parameter Transformer. It used 12 layers, 512 dimensions, and 8 attention heads. Text was tokenized using lower-cased byte pair encoding with a vocabulary of 49,152 items. That architecture mirrors GPT-2: decoder-only, with causally-masked self-attention. ALIGN, a comparable model from Google researchers, used BERT for its text encoder and EfficientNet for its image encoder. The final linear projection at the output of the text encoder takes its dimension from whichever image encoder it is paired with.

  111213148

• 4. Training on the Web

  ›

  WebImageText, the corpus OpenAI assembled for the original CLIP training, contained 400 million image-caption pairs scraped from the internet. Its construction began with every English word appearing at least 100 times in English Wikipedia. Those words were extended with high-mutual-information bigrams, all Wikipedia article titles above a certain search volume, and WordNet synsets. The process produced 500,000 text-queries; each query retrieved up to 20,000 image-text pairs. The total volume of text in WebImageText was comparable to the WebText corpus used to train GPT-2. WebText itself contains roughly 40 gigabytes of text data.

  ALIGN assembled over one billion image-text pairs from web images and their alt-tags. Rather than complex multi-step filtering, it applied only frequency-based filtering. LAION took a different approach: it published its training data under names including LAION-400M, LAION-2B, and DataComp-1B, making those datasets openly available. The image preprocessing for CLIP calculated normalization values from WebImageText's own pixel distribution, setting it apart from standard ImageNet preprocessing. The decision of how long to train each model, and at what resolution, would determine which of the resulting encoders performed best.

  118162122

• 5. Training at Scale

  ›

  The largest ResNet model in the original OpenAI CLIP series trained on 592 V100 GPUs for 18 days. The largest ViT model required 256 V100 GPUs and 12 days. Each of the eight original models trained for 32 epochs, covering the full WebImageText dataset that many times from start to finish. All ViT variants trained at 224-by-224 pixel resolution.

  The ViT-L/14 reached the top of the performance rankings through a post-training modification, not purely by size. After standard training at 224-by-224 pixels, it was retrained at 336-by-336 pixel resolution using a technique called FixRes. That boosted version, designated ViT-L/14@336px, outperformed every other model in the original series.

  OpenCLIP, the series built by LAION on publicly available datasets, extended training further still. Its ViT-L/14 variant ran on 384 A100 GPUs and trained on the LAION-2B dataset for 160 epochs. Over that run, the model processed 32 billion total samples. The resulting encoders could be used by other systems without modification, simply by freezing their weights and pairing them with different architectures.

  12322248

• 6. From Search to Generation

  ›

  Zero-shot image classification in CLIP requires no labeled examples for a given category. To classify an image, the model forms a text embedding for each candidate class using the template 'A photo of a {class}'. The class whose embedding is closest to the image's, by dot product, is the predicted label. Cross-modal retrieval extends this approach. A text query retrieves images by embedding similarity; an image query retrieves related text in the same way.

  Flamingo, a multimodal model released by Google DeepMind in 2022, incorporated CLIP's image encoder as a frozen component. The encoder had been trained as a CLIP pair, using BERT as the text encoder and NormalizerFree ResNet F6 as the image encoder. Once that pair was trained, the text encoder was discarded and the image encoder's weights were frozen. Those frozen weights were then combined with a frozen Chinchilla language model, with a set of newly trained parameters bridging the two.

  Stable Diffusion uses CLIP's text encoder to transform text prompts into the embeddings that guide image generation. CLIP can also serve as a gradient signal for generative processes directly, a technique called CLIP guidance. Fine-tuned CLIP models can rank images by aesthetic quality, a useful filter for large datasets. CLIP further enables image captioning by matching text inputs against image embeddings. Aesthetic ranking, captioning, and generative guidance all trace back to a training run that required no task-specific labeled data. The training pairs were assembled from captions already available online, without any dedicated annotation effort.

  12829303