Multimodal learning

• 1. Introduction

  ›

  Multimodal learning was proposed in 2011, at the very start of the deep learning era. Before that point, machines mostly processed one kind of data at a time. This new approach asked a single model to combine text, audio, images, and video together. The goal was a more complete understanding, built from every kind of information at once rather than just one. How does a machine learn that a sentence and a photograph are describing the same scene? Why did an idea borrowed from statistical mechanics in the 1980s end up teaching computers to connect pictures with words? And what happened by 2023 that turned this quiet research idea into something built into products people were suddenly using everywhere?

• 2. One Idea Two Modalities

  ›

  A caption on a photograph often carries information the image alone cannot show, and that gap is what multimodal learning tries to close. Data naturally arrives in different modalities, each carrying its own layer of meaning. Sometimes an image explains something that text struggles to state plainly. If different words keep showing up under similar-looking images, those words likely describe the same thing. Reverse that logic, and a single word attached to very different images may mean the images share the same object. Making sense of a pattern like that requires a model built to hold information from more than one modality at once. That means combining the information rather than keeping it apart. That kind of joint representation borrowed its foundational idea from an entirely different field: statistical mechanics.

• 3. Borrowed From Physics

  ›

  In 1985, Geoffrey Hinton and Terry Sejnowski invented a stochastic neural network called the Boltzmann machine. It behaves as a stochastic, generative version of the earlier Hopfield net. Its name comes from the Boltzmann distribution, a concept drawn from statistical mechanics. Inside the machine, units split into two groups, visible and hidden, each acting like a neuron with a binary on or off output. A general Boltzmann machine allows any unit to connect to any other unit. That flexibility comes at a cost. The computation needed grows exponentially as the machine gets larger, which makes learning impractical. A restricted Boltzmann machine solves this by permitting connections only between the hidden units and the visible units, never within either group.

  That restricted design is also what makes multimodal deep Boltzmann machines possible. One deep Boltzmann machine handles images, a separate one handles text, and the two are joined at a single shared hidden layer placed on top. That combined structure can process and learn from both types of information at the same time. Boltzmann-based models proved that pairing modalities inside one architecture could work. But the field would soon find a faster way to connect images with language directly.

• 4. Matching Pictures To Words

  ›

  CLIP, which stands for Contrastive Language-Image Pretraining, builds a single shared representation of images and their text descriptions. It trains using contrastive objectives, a method that teaches the model to line up an image with its correct caption rather than any other. The result is a model that can correctly match a picture with the caption that actually belongs to it. That single ability to match a picture with the words that belong to it underlies tasks like visual question answering. There, a system answers questions about what it sees in an image.

• 5. Where This Gets Used

  ›

  Search engines built on multimodal learning let a person type a written description and pull back the matching images directly. This capability is known as cross-modal retrieval. The same underlying skill helps search tools that handle multiple types of media, and improves what platforms recommend to their users. In classification tasks, multimodal deep Boltzmann machines outperform older models such as support vector machines and latent Dirichlet allocation. The same architecture can also predict data missing from a multimodal dataset, such as an image or a passage of text that was never supplied.

  In healthcare, these models combine medical imaging, genomic data, and patient records into one diagnostic picture. That combination improves diagnostic accuracy and helps catch disease earlier, particularly in cancer screening. DALL·E takes a written description and turns it directly into an image, a use that benefits creative industries. Multimodal systems are also used for aesthetic ranking, which scores how visually appealing an image is.

  In robotics, multimodal learning blends inputs like speech, vision, and touch so autonomous systems can interact with people more naturally. This same blending supports broader human-computer interaction as well. Combining visual, audio, and text signals lets multimodal systems read emotion, enhancing sentiment analysis. Systems like this are already put to work in customer service, on social media platforms, and in marketing efforts. Each of these applications depends on a single model holding more than one modality at once, a capability that was about to scale up dramatically.

• 6. Everywhere By 2023

  ›

  Since 2023, Google Gemini and GPT-4o have spread into everyday use as large multimodal models. Their arrival gave everyday users something new: increased versatility and a broader grasp of real-world situations. That came from having one model handle several modalities at once. As more of these large multimodal models reach wider use, the boundary between reading, seeing, and listening keeps blurring. It is getting harder for a machine, and a listener, to tell the difference.