How AI Creates Images and Videos: Diffusion, Latents, and Spacetime Patches

AI does not draw pictures. It denoises static toward a target described by your prompt. The dominant technique in 2026 is a latent diffusion model that learns to remove noise from a compressed image representation, guided step by step by a text encoder that turned your prompt into a set of numeric vectors. Video generation uses the same idea across time, generating and denoising short “spacetime patches” that a decoder reassembles into moving frames. GANs, VAEs, and autoregressive transformers exist and get used in specific niches, but latent diffusion plus transformers is the mainstream stack. Understanding the mechanism kills the two common misunderstandings: that AI copies training images (it does not, but the training data still matters) and that AI hallucinations are magic (they are prediction errors with a specific cause).

AI does not store pictures. It stores weights.

An AI image generator is trained on billions of image-caption pairs, LAION, CommonPool, and licensed corpora being the most cited sources. During training the model does not store the images. It stores a set of numeric weights that describe the statistical relationships between prompts and pixel patterns. When you type a prompt, the model uses those weights to generate a new image that fits the statistical description of your text, not to retrieve a stored copy.

That distinction matters for two reasons. Legally, it is the core of the fair-use argument for training. Practically, it explains why prompting for a specific artist’s style produces a plausible imitation and never the exact painting: there is no painting to retrieve.

Where do the training images come from? Public web scrapes are the biggest source (LAION-5B was the largest public dataset until it was pulled offline for containing CSAM material and reconstructed with filtering). Licensed image libraries (Shutterstock, Getty, and similar) supply the commercial models that need clean rights. First-party data (user uploads that consented to training) is a smaller but growing source. The provenance of a model’s training set is the single strongest predictor of what the model will and will not generate cleanly.

How diffusion models actually work

Diffusion is the algorithm behind Stable Diffusion, DALL-E, Midjourney (via a hybrid stack), Imagen, and every mainstream image generator you would name in 2026. It works in three phases.

Forward diffusion. Take a real training image and add a small amount of random noise. Add more noise. Add more, until the image is pure static. The model is trained to reverse this process by looking at each noise level and learning the transformation from “slightly noisier” to “slightly less noisy.” The forward process is deterministic. The point of training is not the noising, it is teaching the model what “broken” looks like at every level of broken.

Reverse diffusion. At inference time, the model is handed pure static and a text prompt. It runs the learned denoising process backward, one step at a time. At each step it predicts what noise to remove to move a little closer to an image that matches the prompt. After 20 to 50 steps (fewer with modern samplers), the static resolves into a coherent picture.

Latent diffusion. The trick that made diffusion cheap enough to run on a consumer GPU: instead of running diffusion on the raw 512x512x3 pixel grid (huge), a separate autoencoder compresses the image into a low-dimensional latent space (small). Diffusion happens in the latent space. The final latent gets decoded back into pixels. This is why Stable Diffusion runs on 8GB of VRAM and DALL-E 2 did not.

How a text prompt becomes an image

Your prompt is not pixels. It is a sequence of tokens. A text encoder (CLIP in early Stable Diffusion; larger transformer encoders in later models) converts those tokens into a sequence of numeric vectors called embeddings. The embeddings encode the meaning of your words, including subtle relationships. “Cat” and “kitten” sit near each other in embedding space. “Cat” and “car” do not.

The diffusion model uses cross-attention to steer the denoiser. At each denoising step, the model asks: given this partially denoised image and this prompt embedding, which noise should I remove to push the image closer to the prompt? The prompt embedding is not a filter applied at the end. It is a signal wired into every denoising step, which is why prompts affect composition, colour, mood, and content, not just superficial style.

The same prompt gives different images because the starting noise is random. Two runs with the same prompt and the same seed will produce identical images. Change the seed, change the image. This is where “prompt engineering” bumps into a fundamental property of the process: you can steer the sampler, you cannot fully determine it.

GANs, VAEs, and the older approaches that still exist

GANs (Generative Adversarial Networks), invented in 2014 by Ian Goodfellow’s team, train two networks against each other: a generator that produces images and a discriminator that judges whether an image is real or fake. GANs produce sharp, high-frequency detail and dominated image generation from 2014 through about 2020. They are hard to train, prone to mode collapse (generating a narrow subset of possible images), and less controllable via text prompts than diffusion.

VAEs (Variational Autoencoders) compress images into a latent representation and decode from that latent back to pixels. VAEs alone produce blurry images (they optimise for average, which is boring). Modern latent diffusion uses a VAE as the compressor sitting under the diffusion process.

Autoregressive transformers treat an image as a sequence of tokens (a bit like text tokens) and generate one token at a time. Slower per image, but easier to combine with text generation for multimodal models. Google’s Parti and OpenAI’s newer multimodal models use variants of this.

For anyone who wants the vocabulary, what are tokens in AI covers the token concept in more detail, and the AI glossary defines diffusion, latent space, cross-attention, and the rest of the terms.

Why transformers took over image generation too

Early diffusion models used a U-Net (convolutional encoder-decoder) as the core denoiser. That was the standard from 2021 to 2023. In 2024 and 2025, the field moved from U-Nets to Diffusion Transformers (DiTs). Transformers scale better with more compute, generalise better to different aspect ratios and resolutions, and combine more naturally with language models.

Sora, Imagen 3, Stable Diffusion 3, and Flux all use transformer-based denoisers. The convolutional U-Net still exists in older or resource-constrained models, but the frontier moved.

How AI video generation works

Video is exponentially harder than an image because you have to maintain temporal consistency: the cat in frame 30 has to be the same cat as the cat in frame 1, in the same lighting, doing something plausible. A single-frame diffusion approach applied naively to video produces visually stunning individual frames that flicker and morph unnaturally between them.

The 2024-2026 breakthrough was Sora’s spacetime patches approach. Instead of treating a video as a sequence of full frames, Sora tokenises the video into small 3D patches (blocks of pixels across space and a few frames of time). The diffusion model works on these patches directly. Because time is baked into the tokenisation, temporal consistency emerges from the same denoising process that handles space.

Similar architectures now underlie Veo, Runway Gen-3, Kling, and the frontier video models. Adding sound is a separate stack (audio diffusion or codec-token generation) synchronised to the visual output. Image-to-video generation uses the same denoiser starting from a partially-provided latent instead of pure noise.

Honest limits in 2026. Video length caps around 60-90 seconds for coherent output on frontier models. Physics is often subtly wrong (water not obeying gravity, cloth not deforming right, cause-and-effect broken across cuts). Complex scenes with multiple interacting subjects still degrade. Realistic human faces at close range still fall into uncanny territory. The technology is real. The “any video you can describe” claim is not.

What AI can and cannot make reliably

Reliable. Product photography, illustration in named styles, concept art, food photography, portraits at medium distance, textures and materials, environments and landscapes, motion graphics, most short animation sequences, first-draft video for storyboarding.

Unreliable. Hands and fingers (still), text and typography inside images (has improved but still failure-prone), long chains of physical interaction, very specific real-world locations, faces of specific real people at high fidelity (without a fine-tuned LoRA), sports and precise action, dialogue synchronisation with visible mouth movement.

The hands problem is instructive. Hands have high anatomical variance, appear in an enormous number of poses, and are relatively small in training images compared to faces. The model has less signal to learn from and a harder target to hit. Text inside images fails for the same reason: character-level accuracy requires character-level supervision the diffusion process does not naturally provide.

The copyright and commercial-use question

AI-generated images and videos have an unsettled legal status. In the US, the Copyright Office has ruled that purely AI-generated content is not eligible for copyright because it lacks human authorship. Content where a human made significant creative choices (composition, editing, curation, prompt design that meets some threshold) may qualify, with the AI-generated portions still uncopyrightable.

Commercial use is a separate question from copyright. Most model providers grant you commercial rights to what you generate under their terms of service, but those terms often exclude use cases involving identifiable real people, brand assets, or content that could be interpreted as trained on infringing sources. If the commercial use matters, read the current terms of the specific provider on the day you use it. Legal is moving fast.

The other risk vector is likeness. Generating an image that resembles a specific real person, especially a public figure or a celebrity, opens rights-of-publicity claims regardless of who trained what. That risk sits on the user, not the model provider.

How to get better results now that you know the process

Prompt for concept, style, and specific details in that order. The denoiser needs a strong overall signal early and refinement details later. “Photorealistic portrait of a middle-aged woman, side lighting, 85mm lens, muted colour palette” outperforms “Woman standing there.”

Use negative prompts. Most diffusion models accept a “do not include” prompt. It works. Common wins: negative-prompt away “extra fingers”, “text”, “watermark”, “low quality.”

Increase steps for complex scenes. More denoising steps let the model refine complex compositions. Diminishing returns above 50 for most models. Below 20 for anything intricate produces obvious artefacts.

Fine-tune for style consistency. A LoRA (Low-Rank Adaptation) trained on 15-30 examples of your target style gives you consistent output across many prompts without full retraining. LoRAs are the actual answer to “how do I get a consistent character across images.”

Seed-lock what works. When a run produces a good image, save the seed. Small prompt changes with the same seed give controlled variations. Different seeds give different starting noise and are wildly different images.

Do not fight the model on what it cannot do. Hands, text, and specific-person likeness are known failure modes. Correcting them with prompt magic wastes time. Editing the flawed output in a normal image tool is usually faster than re-rolling.

For the tools themselves

For the vetted list of image and video generators, best free AI image generators covers 60 free tools ranked by real traffic and tested, and best free AI video generators does the same on the video side. For the retrieval side of what makes AI answers work in general, how AI search engines work covers the mechanics under the modern multimodal models.

The mechanism, restated once: predict noise to remove, one step at a time, guided by an embedding of your words, decoded from a compressed latent into pixels. Everything else in the current image and video wave is engineering around that core loop. It is prediction, not magic. Once you see it that way, the failure modes stop being surprising and the successes stop being mystical.