Stable Diffusion

Stable Diffusion is a deep learning text-to-image model released in August 2022 by Stability AI, built on diffusion techniques. It is the premier open-weight generative image model and is widely considered a landmark in the AI boom of the 2020s. Unlike proprietary models like DALL-E and Midjourney, Stable Diffusion's code and model weights were released publicly, allowing it to run on consumer hardware with as little as 2.4 GB VRAM.

The model is primarily used for generating detailed images from text descriptions, but it also supports inpainting, outpainting, and image-to-image translation. Its development involved researchers from the CompVis Group at LMU Munich and Runway, with a computational donation from Stability AI and training data from the non-profit LAION. By 2026, Stable Diffusion had evolved through multiple major versions, with SD 3.5 introducing a completely new Rectified Flow Transformer architecture.

Stable Diffusion originated from a project called Latent Diffusion, developed in Germany by researchers at LMU Munich and Heidelberg University. Four of the original five authors (Robin Rombach, Andreas Blattmann, Patrick Esser, and Dominik Lorenz) later joined Stability AI and released subsequent versions.

The original Stable Diffusion (SD 1.4) was released on August 22, 2022. The technical license was released by the CompVis group at LMU Munich, and the model weights were publicly released shortly after. This marked a dramatic departure from previous proprietary text-to-image models like DALL-E and Midjourney, which were only accessible via cloud services.

SD 1.5 (October 2022), SD 2.0/2.1 (November-December 2022), and SD XL (July 2023) followed, each improving image quality and adding new features. SD 3.0 (June 2024) was a major architectural overhaul, replacing the U-Net backbone with a Rectified Flow Transformer. SD 3.5 (October 2024) further refined this architecture and remains the latest official release.

Stable Diffusion has undergone two distinct architectural phases:

Latent Diffusion (SD 1.x - SD XL)

The original architecture consisted of three core components:

• Variational Autoencoder (VAE) — compresses images from pixel space to a smaller latent space, capturing fundamental semantic meaning
• U-Net — the core denoising backbone, comprising a ResNet structure that iteratively removes noise from the latent representation
• Text Encoder (CLIP) — transforms text prompts into embeddings that guide the denoising process via cross-attention

The original model had 860 million parameters in the U-Net and 123 million in the text encoder — remarkably lightweight by modern standards. SD XL scaled this up with a larger U-Net, two text encoders instead of one, and multi-aspect ratio training.

Rectified Flow Transformer (SD 3.0+)

SD 3.0 completely replaced the U-Net with a Rectified Flow Transformer (MMDiT) architecture. The Multimodal Diffusion Transformer mixes text and image encodings inside its operations — unlike previous DiT models where text only conditioned the image encoding unidirectionally. This architectural shift enabled significantly better prompt adherence and image quality, though at the cost of higher computational requirements.

Version	Date	Architecture	Key Features
SD 1.4	Aug 2022	LDM + U-Net	Original release, 860M params
SD 1.5	Oct 2022	LDM + U-Net	Improved training, became community standard
SD 2.0	Nov 2022	LDM + U-Net	New CLIP encoder, upscaler, depth2img
SD XL	Jul 2023	LDM + U-Net	Larger UNet, dual text encoders, Refiner model
SD 3.0	Jun 2024	Rectified Flow Transformer	MMDiT architecture, major quality leap
SD 3.5	Oct 2024	Rectified Flow Transformer	Latest official release, refined MMDiT

Stable Diffusion supports a wide range of image generation and manipulation tasks:

• Text-to-Image — generate images from natural language descriptions
• Image-to-Image — transform an existing image guided by a text prompt
• Inpainting — fill in or replace specific areas of an image
• Outpainting — extend an image beyond its original boundaries
• ControlNet — add conditional control (pose, depth, edge maps, etc.) for precise generation
• Super-Resolution — upscale images while adding detail
• Video Generation — through extensions like Stable Video Diffusion and AnimateDiff

Several popular UIs have been built around Stable Diffusion:

• ComfyUI — node-based workflow interface, popular among power users and professionals
• Stable Diffusion WebUI (AUTOMATIC1111) — the original and most widely used interface
• Stability AI's Platform — official cloud-based generation interface
• Forge — optimized fork of AUTOMATIC1111 with better performance
• InvokeAI — professional-grade open-source interface with canvas-based editing

Aspect	Stable Diffusion	Midjourney	DALL-E 3
Developer	Stability AI	Midjourney	OpenAI
Open Source	Yes (open weights)	No	No
Local Run	Yes (consumer GPU)	Cloud only (Discord)	Cloud only (ChatGPT)
Fine-Tuning	LoRA, DreamBooth, full fine-tune	No (Style reference only)	No
Image Quality	Good (varies by model)	Excellent (stylized)	Excellent (photorealistic)
Cost	Free (open source)	$10-120/month	$20/month (ChatGPT Plus)
Control	Full (ControlNet, IP-Adapter)	Limited	Moderate

Stable Diffusion uses the Stability AI Community License, which permits most uses but restricts certain commercial applications. The model's training data — sourced from LAION-5B, a dataset of billions of images scraped from the web — has been the subject of significant legal controversy. Stability AI faced multiple lawsuits:

• Getty Images v. Stability AI — Getty sued for copyright infringement over training on its watermark-protected images
• Andersen et al. v. Stability AI — class action lawsuit from artists alleging copyright violations
• Multiple ongoing cases regarding the legality of training AI models on publicly scraped data

Stable Diffusion fundamentally changed the landscape of generative AI by proving that high-quality text-to-image models could be open, run on consumer hardware, and be customized by anyone. While its image quality has been surpassed by proprietary models like Midjourney and DALL-E 3 for some use cases, its open-weight nature creates an unmatched ecosystem of community models, tools, and workflows. The shift to Rectified Flow Transformers in SD 3.0+ represents a major architectural evolution. For developers, artists, and researchers who need control, customization, and local execution, Stable Diffusion remains the definitive open-source image generation platform.