DeepDream

• 1. Introduction

  ›

  DeepDream is a computer vision program made by Google engineer Alexander Mordvintsev. It turns a convolutional neural network loose on an ordinary photograph, coaxing out patterns the network was never asked to find. Google called this process algorithmic pareidolia, the tendency to see a face or a familiar shape where none was intended. That tendency, pushed far enough, produces a deliberately overprocessed image with a dream-like appearance, one researchers have compared to a psychedelic experience. Google's program did more than produce strange pictures. It popularized the idea of 'dreaming' as a technical term, describing the generation of images meant to trigger specific activations inside a trained neural network. How does a system trained to classify pictures end up hallucinating inside them? And why would scientists later use brain scans to study whether it resembles a hallucinogenic drug experience?

• 2. Born From Inception

  ›

  Inside Google, the network that would become DeepDream carried the codename Inception, borrowed from the film of the same name. Google's engineers built it for the ImageNet Large-Scale Visual Recognition Challenge, a 2014 competition known as ILSVRC. They released the finished software in July 2015. Yet the underlying idea reaches back to the early history of neural networks, when researchers used similar methods to synthesize visual textures. Several research groups had already developed related visualization ideas before Google published its own.

  Once Google published its techniques and made the underlying code open source, outside developers built their own tools around it. Web services, mobile applications, and desktop software all appeared, letting ordinary users feed in their own photographs and watch the network transform them. That published code exposed the very mechanism at DeepDream's core: a network that could be run in reverse.

• 3. Running The Network In Reverse

  ›

  Ask the network to make a photograph 'more cat-like,' and it does not redraw the image just once. It feeds each adjusted version back in as the next starting point, repeating the process again and again. Researchers have compared the effect to searching a cloud for the outline of an animal or another pattern. This works because the network can run in two directions. Trained originally to classify images by spotting faces and other patterns, the network can also run in reverse. It can be asked to adjust an image so a chosen output neuron reports a higher confidence score. That neuron might be tuned to faces, or to certain animals. This reversal is what lets researchers visualize the emergent structure hidden inside the network, and it is the basis of the DeepDream concept.

  A one-to-many mapping keeps the reversal from ever being perfectly clean or unambiguous. Run the same process through enough repetitions, and even an image that started without the target pattern grows one anyway. That is the pareidolia effect taking hold. The optimization resembles backpropagation, the standard method used to train neural networks. Here, though, the weights stay fixed and the input image is what changes instead.

  Applying gradient descent to each pixel independently leaves neighboring pixels barely related to each other, flooding the image with high-frequency noise. A regularizer that favors natural image statistics, or simple smoothness, cleans the picture up considerably. Mahendran and colleagues used what is called a total variation regularizer, which favors images built from flat, piecewise-constant regions. Yosinski and colleagues discuss further regularizer options, and a later, more visual exploration of these feature-visualization techniques has since been published. That resemblance to hallucinogenic imagery is what led some researchers to compare the network's inner layers directly to the human visual cortex.

• 4. Echoes Of The Visual Cortex

  ›

  Researchers have noted that DeepDream's imagery resembles the hallucinations produced by LSD and psilocybin. They treat that resemblance as a clue to a functional link between artificial neural networks and particular layers of the visual cortex. Neural networks like DeepDream offer a biological analogy that helps researchers understand brain processing and the formation of consciousness. DMT, a hallucinogen, alters the function of the serotonergic system, which is present within the layers of the visual cortex.

  Input vectors are what train a neural network, and internal variations introduced during that training are what alter it. The theory maps those two channels onto the visual cortex: input changes stand in for exogenous signals, and internal changes stand in for endogenous ones. As those internal variations shift inside a deep network, the output image changes to match. That specific manipulation is offered as evidence that inner brain mechanisms work much like the internal layers of a neural network. Turning up the network's internal noise mirrors how hallucinogens block external sensory information, letting a person's preconceived ideas take over their perception instead. That trick of amplifying a network's interior soon spread beyond neuroscience, turning up in a music video and in studies of art history.

• 5. Dreaming Put To Work

  ›

  Dreaming does not stop at a network's output neurons; it can also target the hidden neurons buried inside its internal layers. That exploration reveals what roles and representations different parts of the network have learned. Researchers can also optimize the input for a single neuron, a method sometimes called Activity Maximization, or for an entire layer at once. Dreaming is used most often to visualize a network or to produce computer art. One recent proposal goes further, suggesting that adding dreamed images to a training set could speed up how a network learns abstractions in computer science.

  DeepDream has also been put to work in the field of art history. DeepDream was used for Foster the People's music video for their song 'Doing It for the Money.' That same 'dreamed' visual language would soon be piped into virtual reality, as scientists tested whether it could recreate an altered state of mind.

• 6. Measuring The Hallucination

  ›

  In 2017, a research group at the University of Sussex built a Hallucination Machine, applying the DeepDream algorithm to a pre-recorded panoramic video. The device let users explore virtual reality environments built to mimic the experience of psychoactive substances or psychopathological conditions. The researchers found that the subjective experiences it produced differed significantly from control, non-hallucinogenic videos. Yet those same experiences shared phenomenological similarities with the psychedelic state that follows a dose of psilocybin.

  In 2021, a study in the journal Entropy set out to test the resemblance to psychedelic experience with neuroscientific evidence. The authors recorded electroencephalography, or EEG, from participants while they passively watched a movie clip and its DeepDream-generated counterpart. The DeepDream footage triggered higher entropy in the EEG signal and higher functional connectivity between brain areas, both recognized biomarkers of an actual psychedelic experience.

  In 2022, a research group coordinated by the University of Trento measured participants' cognitive flexibility and creativity. They tested this after exposing participants to virtual reality panoramic videos and to DeepDream-generated hallucinatory counterparts of those same videos. The researchers reported that individuals showed 'an attenuated contribution of the automatic process and chaotic dynamics underlying their decision processes' after the simulated psychedelic exposure. They attributed this presumably to a reorganization in cognitive dynamics that opens the door to uncommon decision strategies and holds back automatic ones. That expanding research follows a name that has outgrown its origin: 'dreaming' now refers to a whole collection of related techniques, not just DeepDream itself.