Stable Diffusion

• 1. Introduction

  ›

  Stable Diffusion arrived publicly in August 2022 with one property that separated it from every rival. Its code and model weights were free to download. The other leading text-to-image tools of that moment required users to generate images through cloud services. This one could run locally, on hardware most people already owned, without paying a subscription. The researchers who built it came from German university labs. The nonprofit that assembled its training data pulled from a vast archive of the public web. Almost immediately, the model's openness became the subject of lawsuits in two countries. The central argument was whether training an AI on billions of scraped images amounted to infringement. Those disputes raised a question the technology had outrun. When a machine learns from the work of human artists without their knowledge, who owns what it produces?

  8997788285

• 2. Origins in Munich

  ›

  The technology underlying Stable Diffusion traces back to a project called Latent Diffusion. Researchers at LMU Munich and Heidelberg University developed it in Germany. Five authors produced the original work. Four of them, Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz, later joined Stability AI and released subsequent versions of the model. Development was led by Rombach, of the CompVis group at LMU Munich, and Esser, of a company called Runway. Both had been among the inventors of the latent diffusion model architecture that Stable Diffusion would use.

  Diffusion models as a class had existed since 2015. Their name came from thermodynamics, which had inspired their earliest designs. The CompVis group developed a refinement called the latent diffusion model in 2021. Where earlier diffusion models worked directly on image pixels, this variant first compressed images into a smaller representational space. That compression made training and generation more efficient, which researchers identified as a key advantage.

  Stability AI provided a computational donation to the project. The company also credited two external organizations as supporters: EleutherAI and LAION, a German nonprofit that assembled the training data. The technical license for the model was released by the CompVis group at LMU Munich. The research paper describing the latent diffusion method was first published in 2021 and updated in 2022. It forms the backbone of the Stable Diffusion architecture.

  5671096

• 3. Five Billion Images

  ›

  LAION-5B, the dataset used to train Stable Diffusion, contained five billion image-text pairs drawn from Common Crawl data scraped from the web. LAION, the German nonprofit that assembled it, received funding from Stability AI. The pairs were filtered by language, by resolution, by the estimated probability of carrying a watermark, and by a predicted aesthetic score. A third-party analysis of a twelve-million-image sample found that approximately 47% of those images came from just 100 different domains. Pinterest alone accounted for 8.5% of that sample. Other large sources included WordPress, Blogspot, Flickr, DeviantArt, and Wikimedia Commons.

  Three subsets of LAION-5B formed the training basis. The final rounds focused on a subset called laion-aesthetics v2 5+, containing around 600 million captioned images. Each image in that subset had been evaluated by a tool called the LAION-Aesthetics Predictor V2. That tool predicted that a human rater would give the image a score of at least 5 out of 10. Images likely to carry watermarks and low-resolution images were excluded. During the final rounds, 10% of text conditioning was randomly dropped to improve a technique called Classifier-Free Diffusion Guidance. The training ran on 256 Nvidia A100 GPUs through Amazon Web Services, consuming 150,000 GPU-hours at a cost of $600,000.

  An investigation by Bayerischer Rundfunk found that LAION's datasets, hosted on Hugging Face, contained large amounts of private and sensitive data.

  2024232526

• 4. From Text to Image

  ›

  Stable Diffusion's U-Net, the component that performs the core denoising work, contains 860 million parameters. The text encoder carries 123 million. By 2022 standards, those figures made the model relatively lightweight. An optimized version could run on a consumer graphics card with as little as 2.4 gigabytes of VRAM. A version built on OpenVINO ran on a CPU alone.

  The primary generation mode is called txt2img. A user writes a text prompt; the model produces an image. Each generation involves a seed value, and reusing the same seed reproduces the same result. A guidance scale parameter controls how closely the output adheres to the prompt; higher values produce more literal interpretations. Generated images are tagged with an invisible digital watermark, though that watermark loses its effectiveness if the image is resized or rotated.

  A second script, called img2img, takes both a text prompt and an existing image as inputs. A strength value between 0.0 and 1.0 determines how much the model modifies the original. Inpainting let users paint a mask over a specific region; the model filled that area with new content based on the prompt. Outpainting extended an image beyond its original edges. A depth-guided extension called depth2img was released on the 24th of November 2022. It inferred the spatial depth of the input image to maintain coherence in the generated result. Researchers have also experimented with img2img for data anonymization, image upscaling, and image compression. The compression approach faces limitations in preserving small text and faces compared to JPEG and WebP.

  Initial training on 512-by-512 pixel images meant output quality degraded noticeably when users specified other dimensions. Human limbs and faces were poorly represented in the training data, and requests for full human figures often produced distorted hands. The model also could not generate legible ambigrams or most other forms of text and typography. Fine-tuning for anime character generation required a minimum of 30 gigabytes of VRAM. Cards in Nvidia's GeForce 30 series typically provide only about 12 gigabytes.

  1614395040

• 5. Copyright and Consent

  ›

  Sarah Andersen, Kelly McKernan, and Karla Ortiz filed a copyright infringement lawsuit in January 2023. They sued Stability AI, Midjourney, and DeviantArt. Their claim was that these companies had infringed the rights of millions of artists. AI training tools had used images scraped from the web without the consent of the people who created them.

  That same month, Getty Images initiated legal proceedings against Stability AI in the English High Court. Getty claimed that Stability AI had scraped millions of its images without authorization and used them to train Stable Diffusion. One key dispute concerned where the training had taken place. Stability AI argued that all training and development occurred in U.S. data centers operated by Amazon Web Services. Getty also argued that under UK copyright law, making the pre-trained software available in the UK through platforms including GitHub, HuggingFace, and DreamStudio could constitute secondary infringement.

  In July 2023, U.S. District Judge William Orrick indicated he was inclined to dismiss most of the Andersen, McKernan, and Ortiz suit. He gave the plaintiffs an opportunity to file a revised complaint. The Getty case proceeded more slowly. As of the 4th of November 2025, Getty had largely lost the lawsuit. The outcome prompted Getty and some lawyers to call for stronger copyright protections in Britain.

  Stable Diffusion was more permissive than competing commercial products in the types of content users could generate, including violent and sexually explicit imagery. Stability AI's CEO, Emad Mostaque, described responsible use as "peoples' responsibility." He argued that open access would produce a net benefit for society despite potential harms. He also argued that the open release was intended to end corporate control over AI image generation. Any restrictions Stability AI placed on content could be bypassed through the publicly available source code. Controversy arose around photorealistic sexualized depictions of minors, with such images appearing on platforms including Pixiv. In June 2024, attackers hacked an extension of ComfyUI and targeted users they accused of committing "one of our sins." Their stated list included AI-art generation.

  8283858881

• 6. Fine-Tuning and Extensions

  ›

  In 2022, researchers from Google Research and Boston University developed DreamBooth. It allowed users to fine-tune the model toward a specific subject using a small set of reference images. The result was a version of the model capable of generating that subject in new contexts and styles.

  A technique called textual inversion came from researchers at Tel Aviv University in 2022, with support from Nvidia. It trained an "embedding": a vector representation linked to a new pseudo-word. Using that word in a prompt would reliably produce a consistent visual concept. Embeddings could reduce biases in the original model or capture a particular visual style.

  A developer known as Kurumuz, who worked at NovelAI, created hypernetworks in 2021 as a technique originally intended for text-generation models. Hypernetworks are small pretrained neural networks applied to a larger model at specific points. They identify areas of visual importance, such as hair and eyes. Adjustments in a secondary latent space can steer the output toward a chosen style.

  ControlNet's defining feature was a split of a neural network's weights into a locked copy and a trainable copy. The locked copy preserved the original model; the trainable copy learned a new controlling condition. ControlNet used a technique called zero convolution, in which every weight and bias began at zero. This prevented new training from distorting the underlying model before it had learned anything. Common controlling inputs included depth maps, edge maps, and skeletal pose diagrams.

  ControlNet's design enabled fine-tuning to run on personal devices with small datasets. Stability AI used the same principle to release a dedicated inpainting model.

  4151374243

• 7. Interfaces and Releases

  ›

  DreamStudio, Stability AI's cloud-based image-generation service, supported negative prompts that let users specify elements the model should avoid generating. Stability AI also published an open source version called StableStudio. Among third-party interfaces, AUTOMATIC1111 became the most widely used, offering additional features beyond the core model. Fooocus aimed to reduce the amount of prompting required. ComfyUI built the generation pipeline as a node-based visual interface, comparable in structure to software used in three-dimensional modeling.

  The model launched without a version 1.0. Versions 1.1 through 1.4 were released as an initial group by CompVis. Version 1.5 followed on the 20th of October 2022, released by RunwayML and initialized with the weights of version 1.2. Version 2.0 arrived in November 2022, retrained from scratch on a filtered dataset. Stable Diffusion XL 1.0 reached users in July 2023 with 3.5 billion parameters, roughly 3.5 times more than earlier versions. XL Turbo followed in November 2023, distilled from XL 1.0 to run in fewer diffusion steps. Stable Diffusion 3.0 arrived in early preview in February 2024, replacing the U-Net backbone with a Rectified Flow Transformer. Stable Diffusion 3.5 followed in October 2024 as a family of models. Large carried 8 billion parameters, Medium carried 2.5 billion, and Large Turbo was distilled from Large for faster generation.

  Prior to Stable Diffusion 3, the model applied a license called Creative ML OpenRAIL-M. That license prohibited uses including crime, harassment, doxing, and providing medical advice. Stable Diffusion 3.5 introduced a Community License for most users. Commercial organizations with annual revenue above one million dollars required a separate Enterprise License. Users retain the rights to their generated images and may use them commercially under either license. Stable Diffusion 3.5 Large became available through Amazon Bedrock for enterprise use. The training runs that built the original model had also run on Amazon Web Services.

  5554595857