Model architecture field guide

Generative Model Architectures

Understand what actually sits behind text, embedding, image, video, audio, and multimodal products—then choose a backbone, representation, generator, and system wrapper deliberately.

Start with a use case Explore all 22 families

The essential distinction

A capability is not an architecture

“Chat,” “image,” and “multimodal” describe an interface. Decoder-only, state-space, DiT, and MoE describe internal computation. RAG and tool use describe the runtime system around a model. One production system can legitimately carry all three kinds of label.

Customer-support copilot
= chat capability
+ decoder-only backbone
+ retrieval and reranking
+ typed tool controller
+ policy and human escalation

Write an architecture passport

1Capability: Inputs, outputs, user job, and required evidence
2Representation: Tokens, vectors, pixels, codec codes, or continuous latents
3Backbone: Encoder, decoder, seq2seq, attention, SSM, or hybrid
4Capacity: Dense or expert-routed; total versus active parameters
5Generation: Autoregressive, diffusion/flow, adversarial, or deterministic
6System: Retrieval, tools, validators, permissions, monitoring, and humans

Model-selection matrix

Start with the user job and failure mode. These are baseline architectures to test—not universal winners.

Product need	Test this baseline first	Why it fits	Watch closely
Open-ended assistant or coding copilot	Decoder-only LM + retrieval/tools as needed	Flexible generation and in-context task adaptation	Grounding, prompt injection, tool authority, serial latency
Classification, extraction, or reranking at volume	Encoder-only model or compact encoder–decoder	One-pass bounded prediction can be cheaper and easier to calibrate	Domain shift, input length, label calibration
Translation, transcription, or source-faithful transformation	Encoder–decoder model	Dedicated source memory conditions every generated token	Omissions, names/numbers, source length, decode latency
Semantic search over a large corpus	Dual encoder + vector index + optional reranker	Candidates can be embedded once and searched efficiently	Recall, negatives, filters, score calibration
Answers from current or private knowledge	Hybrid RAG + reranker + grounded generator	Knowledge remains external, inspectable, and refreshable	Ingestion, access control, missing evidence, poisoned context
Image creation or editing	Latent diffusion with U-Net or DiT denoiser	Compressed iterative generation supports strong conditioning and edits	Codec detail loss, sampling latency, rights and provenance
Video creation	Spatiotemporal latent diffusion or media-token generator	Both visual detail and temporal dependence are modeled	Identity drift, motion, duration, compute, audio sync
Speech recognition	Audio encoder–decoder designed for ASR	Separates acoustic understanding from text generation	Language, accent, overlap, noise, streaming delay
Long streaming sequence under a tight state budget	SSM/recurrent or attention–SSM hybrid	Carries compact recurrent state instead of a full attention cache	Exact long-range recall, kernel support, real throughput
Multimodal document or chart assistant	Vision encoder + language decoder + document retrieval	Preserves visual evidence while grounding answers in source pages	Small text, spatial references, media token cost, citations
Reliable actions in business systems	Tool-using LM inside a deterministic policy/controller	Language handles intent while code enforces permissions and side effects	Idempotency, approvals, retries, schemas, audit trail

Architecture explorer

Search by task, mechanism, strength, or failure mode. Open any family for training, inference, and selection guidance.

Search the taxonomy

Showing 22 of 22 architecture families

Product capabilities

What the product accepts and returns. These are deployment shapes, not mutually exclusive neural architectures.

Capability · generative text

Text & chat language models

#text-chat-language-models

Generate text one token at a time, usually with a causal decoder trained for next-token prediction and then adapted to follow instructions.

Mental model

A probabilistic continuation engine wrapped in a conversation protocol—not a database, search engine, or deterministic rules engine.

Data flow

Messages, documents, or code
Tokenizer + role/tool markers
Causal language-model backbone
Next-token distribution
Decode until a stop condition

Open the full decision guide

How it trains

Pretraining commonly minimizes next-token cross-entropy over large token sequences. Instruction tuning, preference optimization, safety training, and tool-use examples then shape interaction behavior; they do not change the basic need for evidence and evaluation.

How inference runs

The model repeatedly predicts and samples or selects one next token. A KV cache reuses prior attention state; long answers remain serial, and decoding settings change variability rather than factuality.

Strengths

Open-ended writing, transformation, explanation, code, and conversation
Learns tasks from instructions and examples in context
Can emit structured tool calls or schemas when constrained and validated

Trade-offs

Can produce fluent unsupported claims
Autoregressive output adds per-token latency
Context, prompting, and sampling choices materially affect behavior

Use it when

The task needs flexible language generation or synthesis
A rubric and representative evaluation set can define acceptable behavior
Fresh facts can be grounded through retrieval or tools

Avoid or challenge it when

A deterministic parser, query, or rules engine solves the task reliably
Exact current facts are required but no trusted source is connected
Unreviewed output could directly trigger a high-impact action

Illustrative published families

• GPT-style causal language models
• Instruction-tuned descendants such as the InstructGPT research system

Commonly combines with

Decoder-only TransformerMixture of ExpertsRetrieval-augmented systemsTool & reasoning systems

Primary referencesAttention Is All You Need Language Models are Few-Shot Learners InstructGPT

Capability · vector representation

Embedding models

#embedding-models

Map text, images, audio, users, or items into vectors whose geometry is trained to preserve a useful notion of similarity.

Mental model

A learned coordinate system. Nearness means “similar under the training objective,” not universally equivalent or factually related.

Data flow

Input item
Encoder
Pooling or projection
Fixed-length vector
Similarity search, clustering, or classifier

Open the full decision guide

How it trains

Contrastive, metric-learning, classification, or paired-data objectives pull useful matches together and push negatives apart. The negative-sampling strategy and domain data define what similarity means.

How inference runs

Each item is encoded in one forward pass. Stored vectors enable fast approximate-nearest-neighbor search; a cross-encoder can rerank the small candidate set for finer interaction.

Strengths

Efficient semantic retrieval over large collections
Reusable features for clustering, routing, recommendations, and deduplication
Candidate vectors can be computed and indexed ahead of time

Trade-offs

A single vector compresses away token-level detail
Similarity degrades under domain, language, or time drift
Scores are model- and index-specific, not calibrated probabilities

Use it when

You need semantic candidate retrieval
The corpus is too large for pairwise scoring
You can evaluate recall on production-shaped queries

Avoid or challenge it when

Exact lexical matching is the only requirement
You need a generated response rather than a representation
A similarity threshold would be deployed without calibration

Illustrative published families

• BERT-derived sentence encoders
• The separate text and image towers used by CLIP

Commonly combines with

Encoder-only TransformerContrastive / dual encodersRetrieval-augmented systems

Primary referencesBERT Sentence-BERT CLIP

Capability · visual synthesis

Image generation & editing

#image-generation

Synthesize or transform pixels, most often through a text or image conditioner, a latent generator, and an image decoder.

Mental model

A pipeline, not one network: prompt encoder → generative process → latent/pixel decoder, often with separate control and safety components.

Data flow

Text, image, mask, or layout condition
Condition encoder
Latent diffusion, DiT, or autoregressive generator
Image decoder
Candidate image + checks

Open the full decision guide

How it trains

Modern systems often learn a denoising or flow objective in pixel or compressed latent space. Alignment to text depends on paired data and conditioning; autoencoders may be trained separately.

How inference runs

A sampler starts from noise or a noised source image, iteratively refines it under the condition, then decodes the latent. Seeds improve reproducibility but do not make all runtime paths deterministic.

Strengths

High-quality open-ended visual synthesis
Natural support for variation, inpainting, and guided editing
Latent pipelines make high-resolution generation more practical

Trade-offs

Multiple sampling steps add latency
Text, typography, identity, and exact spatial constraints need explicit evaluation
Training-data rights, provenance, and misuse controls are product concerns

Use it when

Creative ideation, asset drafts, or controlled image transformation
The output can be reviewed or constrained
Visual quality and prompt adherence are evaluated separately

Avoid or challenge it when

Pixel-exact layouts are required
The output is evidence of a real event or identity
Rights and provenance requirements are unresolved

Illustrative published families

• Latent Diffusion / Stable Diffusion research lineage
• Diffusion Transformer (DiT) research architecture

Commonly combines with

Latent diffusionDiffusion TransformerVision encodersMultimodal fusion

Primary referencesLatent Diffusion Models Diffusion Transformers

Capability · temporal visual synthesis

Video generation

#video-generation

Generate spatial and temporal structure together using frame, patch, or latent video representations with cross-frame computation.

Mental model

Image generation plus a time axis—and therefore a much larger consistency, memory, motion, and evaluation problem.

Data flow

Text, image, or video condition
Spatiotemporal representation
Temporal diffusion or token generator
Video decoder / upsampler
Frames + audio alignment checks

Open the full decision guide

How it trains

Objectives include spatiotemporal denoising in pixel or latent space and next-token prediction over visual codes. Conditioning can include text, key frames, motion, masks, or camera controls.

How inference runs

The system samples a clip jointly or in windows, often with cascaded refinement. Longer duration, higher resolution, and stronger temporal coherence all increase compute and memory pressure.

Strengths

Text- or image-conditioned motion synthesis
Video extension, interpolation, and editing
Latent representations can amortize the cost of raw pixels

Trade-offs

Identity and object persistence can drift over time
Generation and human review are expensive
Physics, timing, readable text, and synchronized audio require dedicated tests

Use it when

Short creative clips or controlled transformations
Temporal failure modes are included in the acceptance rubric
Asynchronous generation is acceptable

Avoid or challenge it when

A real-time deterministic renderer is required
Continuity or physical correctness cannot be reviewed
The workflow cannot support provenance and consent controls

Illustrative published families

• Video Diffusion Models research family
• Latent video diffusion and token-based video generators

Commonly combines with

Latent diffusionDiffusion TransformerAutoregressive media modelsMultimodal fusion

Primary referencesVideo Diffusion Models Diffusion Transformers

Capability · temporal audio

Audio, speech & music models

#audio-speech-music

A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.

Mental model

Choose the task first. ASR maps audio to text; TTS and music generation map conditions to acoustic or codec representations; understanding maps audio to labels or embeddings.

Data flow

Waveform or spectrogram
Acoustic / codec representation
Encoder, seq2seq, diffusion, or token generator
Text, labels, codec tokens, or waveform
Task-specific decoding

Open the full decision guide

How it trains

Objectives vary: transcription uses sequence likelihood, representation learning may mask audio, codec generators predict discrete audio tokens, and diffusion systems learn denoising in waveform, spectrogram, or latent space.

How inference runs

ASR usually encodes audio then decodes text; generation may predict codec tokens autoregressively or iteratively denoise. Streaming constraints can require chunked encoders and bounded look-ahead.

Strengths

Speech transcription and translation
Natural speech, sound, and music synthesis
Audio embeddings support search, moderation, and classification

Trade-offs

Accents, noise, overlap, code-switching, and domain vocabulary shift quality
Long-form generation needs temporal structure and consistency
Voice identity, consent, watermarking, and copyright need explicit governance

Use it when

The modality itself carries needed information
You have task-, language-, and environment-specific evaluations
Latency and streaming requirements are designed up front

Avoid or challenge it when

A transcript already contains all relevant information
Identity cloning lacks informed consent
One benchmark is being used as proof across languages and acoustic conditions

Illustrative published families

• Whisper-style encoder-decoder ASR
• AudioLM-style hierarchical codec-token generation
• Waveform or latent diffusion audio systems

Commonly combines with

Encoder-decoder TransformerAutoregressive media modelsDiffusionContrastive encoders

Primary referencesWhisper AudioLM wav2vec 2.0

Core backbones

How information moves through the trainable network. A deployed model may combine several of these choices.

Backbone · bidirectional context

Encoder-only Transformers

#encoder-only-transformers

Let every input token attend to tokens on both sides, producing contextual representations rather than an open-ended generation loop.

Mental model

Read the entire input, contextualize each position, then pool or attach a task head.

Data flow

Token sequence
Bidirectional self-attention blocks
Contextual token states
Pooler or task head
Class, span, score, or vector

Open the full decision guide

How it trains

Masked-token reconstruction is the classic pretraining objective, followed by task-specific classification, token labeling, span prediction, or metric learning.

How inference runs

One forward pass processes the available input. It is naturally suited to understanding and representation, but does not by itself generate an arbitrary continuation.

Strengths

Rich bidirectional context for classification and extraction
Parallel processing across the input
Often efficient for high-volume bounded tasks

Trade-offs

Not an open-ended generator
Maximum input length and quadratic attention can constrain long documents
A task head or pooling strategy is usually required

Use it when

Classification, named entities, reranking, or embeddings dominate
The complete input is available before prediction
A smaller specialized model can meet the rubric

Avoid or challenge it when

The primary output is long free-form text
Streaming generation is required
Inputs routinely exceed the evaluated context strategy

Illustrative published families

• BERT and RoBERTa research families
• Encoder-based rerankers and classifiers

Commonly combines with

Embedding modelsVision encodersContrastive / dual encodersRetrieval systems

Primary referencesBERT

Backbone · causal generation

Decoder-only Transformers

#decoder-only-transformers

Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.

Mental model

Compress the prefix into attention state, predict one continuation token, append it, and repeat.

Data flow

Prefix tokens
Causally masked self-attention
Hidden state at each position
Vocabulary logits
Next token → repeat

Open the full decision guide

How it trains

Next-token cross-entropy supplies a learning signal at every position. Post-training may add instruction examples, preferences, verifiable rewards, or tool trajectories.

How inference runs

Prefill processes the prompt; decode then emits tokens serially while the KV cache stores past attention keys and values. Context size, cache memory, and output length drive serving behavior.

Strengths

One scalable interface for many generative tasks
Strong in-context adaptation
Can interleave prose, code, structured data, and tool-call tokens

Trade-offs

Serial decode latency
Quadratic prefill in standard full attention
Prompt sensitivity and probabilistic output require validation

Use it when

Open-ended generation or interactive chat is central
Many tasks can share one model endpoint
You can constrain and evaluate outputs in the product loop

Avoid or challenge it when

A small encoder or deterministic component meets the requirement
Guaranteed exact outputs are assumed without validation
Long source corpora should be searched rather than placed wholesale in context

Illustrative published families

• GPT-style causal language models
• LLaMA research family

Commonly combines with

Text & chatMixture of ExpertsMultimodal fusionRAGTool systems

Primary referencesAttention Is All You Need LLaMA

Backbone · sequence-to-sequence

Encoder–decoder Transformers

#encoder-decoder-transformers

Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.

Mental model

A reader and a writer connected by cross-attention.

Data flow

Source sequence
Bidirectional encoder
Source memory
Causal decoder + cross-attention
Target sequence

Open the full decision guide

How it trains

Teacher-forced target-token likelihood is common. Text-to-text denoising or span corruption lets many tasks share the same input/output interface.

How inference runs

The source is encoded once; the decoder generates the target autoregressively while attending to stable source states. Beam search is useful in some bounded tasks but is not universally best.

Strengths

Clear separation between source understanding and target generation
Strong fit for translation, summarization, transcription, and transformation
Source memory is reused throughout decoding

Trade-offs

Two stacks can increase parameters and deployment complexity
Target decoding remains serial
May be less convenient than one decoder-only interface for heterogeneous chat tasks

Use it when

Output is tightly conditioned on a distinct input
Source and target have different structures or modalities
Faithful transformation matters more than open-ended continuation

Avoid or challenge it when

A single general chat interface is the overriding requirement
No distinct source sequence exists
A non-generative encoder head is sufficient

Illustrative published families

• Original Transformer
• T5
• Whisper speech recognition architecture

Commonly combines with

Text transformationAudio / speechMultimodal fusion

Primary referencesAttention Is All You Need T5 Whisper

Backbone modifier · sparse capacity

Mixture of Experts (MoE)

#mixture-of-experts

Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.

Mental model

Large parameter capacity with sparse per-token activation; sparse compute is not the same as small memory or simple serving.

Data flow

Token hidden state
Router scores experts
Top-k expert dispatch
Expert transformations
Weighted merge + residual path

Open the full decision guide

How it trains

The main task loss trains experts and router together. Load-balancing or routing regularizers discourage collapse into a few experts; distributed implementations must coordinate token dispatch.

How inference runs

Each token visits selected experts. Arithmetic can remain sparse while model weights, inter-device communication, routing imbalance, and batching still shape latency and cost.

Strengths

More parameter capacity without activating every parameter per token
Experts may specialize through learned routing
Scales naturally inside Transformer feed-forward blocks

Trade-offs

Large weight memory and distributed communication
Routing imbalance and capacity overflow complicate training
Reported total parameters do not reveal active compute or serving efficiency

Use it when

Large-scale training and serving infrastructure can exploit sparse routing
Capacity is more constrained than per-token arithmetic
You can benchmark the actual hardware topology

Avoid or challenge it when

A single-device or memory-constrained deployment is required
“Sparse” is being assumed to guarantee lower latency
Operational simplicity is more valuable than extra capacity

Illustrative published families

• Switch Transformer
• Sparse expert layers inside language or multimodal models

Commonly combines with

Decoder-only TransformerEncoder-decoder TransformerDiffusion Transformer

Primary referencesSwitch Transformers

Backbone · recurrent state

State-space & recurrent models

#state-space-recurrent-models

Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.

Mental model

Stream the sequence through a learned state transition; the state is a compressed memory of the past.

Data flow

Current token + prior state
Learned state update
Selective gate or recurrent mixer
Output hidden state
Carry state forward

Open the full decision guide

How it trains

Sequence modeling losses are paired with recurrence or structured state-space layers. Parallel scans or related formulations can make training more parallel than a naive step-by-step RNN.

How inference runs

Decode maintains a bounded recurrent state per layer. Hybrid architectures may alternate state-space, convolution, and attention layers to recover capabilities each mechanism handles well.

Strengths

Linear-time sequence processing in the core recurrence
Compact streaming state at inference
Natural fit for long or continuously arriving sequences

Trade-offs

Compressed state can lose precise distant details
Tooling and optimized kernels may be less mature than standard attention
Architecture-level complexity claims must be verified end to end

Use it when

Streaming or long-sequence efficiency is a first-order requirement
The target runtime has optimized kernels
Recall, quality, and throughput are benchmarked on the real workload

Avoid or challenge it when

Exact arbitrary retrieval from long context is assumed without testing
The serving stack cannot exploit the architecture
A mature Transformer deployment already meets the budget

Illustrative published families

• Mamba selective state-space models
• RWKV-style recurrent language models
• Hybrid attention–SSM stacks

Commonly combines with

Decoder language modelsMixture of ExpertsHybrid attention layers

Primary referencesMamba RWKV

Representation & generation

How images, audio, video, and other high-dimensional outputs are represented, learned, and sampled.

Representation · compression

Autoencoders, VAEs & learned tokenizers

#autoencoders-vaes-tokenizers

Learn an encoder that compresses data and a decoder that reconstructs it; variational and quantized variants shape the latent space for sampling or downstream generation.

Mental model

A learned codec. It decides which information survives before another model generates in the compressed space.

Data flow

Image, audio, or other signal
Encoder
Continuous latent or discrete codes
Optional latent generator
Decoder reconstruction

Open the full decision guide

How it trains

A reconstruction objective preserves perceptual content. VAEs add a distribution regularizer; vector-quantized models map representations to a discrete codebook and add codebook/commitment objectives.

How inference runs

Encode for compression or editing; sample or generate a latent/code sequence; decode to the original modality. Reconstruction quality limits the best possible downstream output.

Strengths

Reduces the dimensionality of expensive media
Creates reusable continuous latents or discrete tokens
Separates representation learning from the generative prior

Trade-offs

Compression discards detail
Latent geometry may not match downstream semantics
Quantized codebooks can be underused or introduce artifacts

Use it when

Raw media is too expensive to model directly
A generative model needs a compact continuous or discrete space
Reconstruction failures are measured by content slice

Avoid or challenge it when

Lossless reconstruction is mandatory
The codec domain differs materially from production data
Decoder artifacts would be mistaken for generator failures

Illustrative published families

• Variational Autoencoder (VAE)
• VQ-VAE discrete visual/audio tokenizers
• Autoencoders used by latent diffusion

Commonly combines with

Latent diffusionAutoregressive media modelsDiffusion Transformer

Primary referencesAuto-Encoding Variational Bayes VQ-VAE

Generator · adversarial learning

Generative Adversarial Networks (GANs)

#generative-adversarial-networks

Train a generator to fool a discriminator while the discriminator learns to distinguish generated samples from training data.

Mental model

A forger and a critic improve against each other; after training, the forger can generate in one forward pass.

Data flow

Random latent + condition
Generator
Synthetic sample
Discriminator compares real vs synthetic
Adversarial gradients update both

Open the full decision guide

How it trains

A minimax adversarial objective couples two networks. Practical variants change losses, regularization, normalization, and conditioning to stabilize training and control outputs.

How inference runs

Sample a latent and run the generator once, making generation fast. Inversion or editing requires additional machinery because there is no built-in iterative reverse process.

Strengths

Fast one-pass sampling
Sharp outputs in well-scoped visual domains
Useful conditional and image-to-image variants

Trade-offs

Training instability and mode collapse
Coverage and likelihood are hard to assess
Large open-domain text-conditioned generation has shifted toward other families

Use it when

Low-latency sampling matters in a constrained domain
A proven GAN pipeline already fits the data and controls
Diversity and coverage are explicitly evaluated

Avoid or challenge it when

Stable training and broad mode coverage are primary requirements
The team lacks domain-specific evaluation
A newer diffusion or autoregressive baseline is not being compared

Illustrative published families

• Original GAN formulation
• StyleGAN research family
• Conditional image-to-image GANs

Commonly combines with

Convolutional or Transformer generatorsEncoders for inversion and editing

Primary referencesGenerative Adversarial Networks

Generator · invertible density model

Normalizing flows

#normalizing-flows

Transform a simple distribution into a complex data distribution through a sequence of invertible mappings with tractable Jacobians.

Mental model

A reversible deformation of probability space: data can map to noise and noise can map back to data.

Data flow

Data sample
Invertible transformations
Simple latent distribution
Exact change-of-variables likelihood
Reverse transforms for sampling

Open the full decision guide

How it trains

Maximum likelihood is optimized exactly under architectural constraints that make inversion and the Jacobian determinant tractable.

How inference runs

Density evaluation runs data toward the latent; generation samples the base distribution and applies every transform in reverse.

Strengths

Exact likelihood under the model
Invertible encoding and generation
Useful when density estimation is itself important

Trade-offs

Invertibility constrains network design
High-dimensional media can require deep, memory-heavy flows
Likelihood does not necessarily track perceived sample quality

Use it when

Exact density or reversible transforms are requirements
The domain fits available invertible architectures
You will evaluate both likelihood and task utility

Avoid or challenge it when

Only perceptual generation quality matters
Architectural flexibility is more important than exact likelihood
A simpler discriminative uncertainty method is sufficient

Illustrative published families

• Real NVP
• Glow-style image flows

Commonly combines with

Variational inferenceHybrid latent-variable models

Primary referencesReal NVP

Generator · iterative denoising

Diffusion & score-based generation

#pixel-space-diffusion

Learn to reverse a gradual noising process, producing data by repeatedly transforming noise toward a sample.

Mental model

Start with static and repeatedly remove the predicted noise, guided by a time step and optional condition.

Data flow

Clean training sample + sampled noise level
Noisy sample
U-Net or Transformer denoiser
Noise / velocity / score estimate
Iterative reverse sampler

Open the full decision guide

How it trains

A network predicts noise, clean data, velocity, or a score at randomly sampled noise levels. Related flow-matching formulations learn a vector field between distributions.

How inference runs

Sampling integrates a learned reverse process across multiple steps. Sampler choice, step count, guidance, and seed trade speed, diversity, and fidelity.

Strengths

Stable training relative to adversarial objectives
Flexible conditioning and editing
Strong coverage across image, audio, video, and scientific data

Trade-offs

Iterative sampling is slower than a one-pass generator
Guidance can reduce diversity or introduce artifacts
The data representation and sampler materially affect results

Use it when

High-dimensional conditional generation is central
Multiple inference steps fit the latency budget
Controls and sample quality can be evaluated jointly

Avoid or challenge it when

One-pass generation is mandatory
The model must run on a severely constrained device
“Diffusion” is being treated as one fixed implementation

Illustrative published families

• DDPM
• Score-based models
• Pixel-space diffusion systems

Commonly combines with

U-NetDiffusion TransformerLatent diffusion

Primary referencesDenoising Diffusion Probabilistic Models Score-Based Generative Modeling

Generator · compressed-space denoising

Latent diffusion

#latent-diffusion

Run diffusion in an autoencoder’s lower-dimensional latent space, then decode the generated representation back to pixels or another signal.

Mental model

Denoise a compact learned sketch instead of every raw pixel.

Data flow

Training media
Frozen or jointly trained autoencoder latent
Conditional latent denoiser
Sampled clean latent
Decoder → media

Open the full decision guide

How it trains

An autoencoder first learns a perceptual compression. The diffusion model then learns the denoising objective over those latents, often conditioned through cross-attention.

How inference runs

Sample in the smaller latent tensor over several steps and decode once. Image-to-image and inpainting can begin from encoded and selectively noised source content.

Strengths

Lower denoising cost than raw high-resolution pixels
Modular text conditioning and editing
Reusable autoencoder and generator components

Trade-offs

The autoencoder creates a reconstruction ceiling
Fine text or high-frequency detail can be lost in compression
Two-stage failures are harder to attribute

Use it when

High-resolution media generation must fit practical compute
Editing and conditional control are required
Codec reconstruction is validated on the target domain

Avoid or challenge it when

The autoencoder drops task-critical details
End-to-end simplicity matters more than generation cost
Latent-space shortcuts could hide safety-relevant content

Illustrative published families

• Latent Diffusion Models
• Stable Diffusion research lineage

Commonly combines with

Autoencoder / VAEU-Net denoiserDiffusion TransformerText encoder

Primary referencesLatent Diffusion Models

Generator backbone · Transformer denoiser

Diffusion Transformers (DiT)

#diffusion-transformers

Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.

Mental model

DiT changes the denoiser backbone; it does not eliminate the diffusion process, latent codec, conditioning, or sampler.

Data flow

Noisy latent
Patchify + position / time conditioning
Transformer blocks
Predicted latent update
Sampler step → repeat

Open the full decision guide

How it trains

The model learns the same class of denoising, velocity, or flow objective as other diffusion systems while Transformer blocks mix information between latent patches.

How inference runs

A sampler repeatedly invokes the Transformer at different noise or flow times, then a decoder maps the final latent to media.

Strengths

Transformer scaling and implementation ecosystem
Global interactions between latent patches
Natural path to shared multimodal token processing

Trade-offs

Attention cost grows with latent token count
Still requires repeated denoiser calls
“DiT” alone does not specify codec, objective, conditioning, or sampler

Use it when

Transformer scaling infrastructure is available
Latent patch count and sampling latency fit the budget
You need a flexible backbone for image or video generation

Avoid or challenge it when

A small U-Net already meets the objective
The label is being used as a complete architecture specification
High-resolution token counts overwhelm the target runtime

Illustrative published families

• DiT research family
• Transformer denoisers inside latent image and video pipelines

Commonly combines with

Latent diffusionAutoencoder / VAEMultimodal conditioningMixture of Experts

Primary referencesDiffusion Transformers

Generator · discrete media tokens

Autoregressive visual & audio models

#autoregressive-media-models

Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.

Mental model

Make media look like a language: tokenize it, choose an ordering, predict the next code, then decode.

Data flow

Media tokenizer / codec
Ordered discrete codes
Causal token model
Generated code sequence
Media decoder

Open the full decision guide

How it trains

A learned codec is trained for reconstruction; the prior minimizes next-code cross-entropy, optionally conditioned on text, prior frames, semantic tokens, or coarse-scale codes.

How inference runs

Codes are generated serially or in hierarchical/blockwise schedules, then decoded. Ordering determines latency and which dependencies are easy to model.

Strengths

Unified token-based modeling across modalities
Compatible with causal language-model infrastructure
Exact discrete likelihood for the chosen tokenization

Trade-offs

Long code sequences create serial latency
Tokenizer artifacts bound output quality
A one-dimensional ordering can be awkward for spatial or multiscale structure

Use it when

A strong discrete codec exists
Cross-modal token modeling or continuation is important
Hierarchical generation can meet latency goals

Avoid or challenge it when

Long serial decoding misses the budget
The codec loses critical perceptual detail
A continuous diffusion representation is substantially simpler

Illustrative published families

• ImageGPT-style pixel/token modeling
• VQ-token image generators
• AudioLM-style codec-token hierarchy

Commonly combines with

Decoder-only TransformerVQ-VAE / neural codecMultimodal token fusion

Primary referencesImageGPT AudioLM

Modality interfaces

How vision, text, audio, and other modalities are encoded, aligned, fused, or connected to a generator.

Modality interface · visual representation

Vision encoders

#vision-encoders

Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.

Mental model

A visual reader. It creates representations; generation requires a decoder or generative model.

Data flow

Image or frames
Patch, convolutional, or hybrid stem
Vision backbone
Spatial tokens / feature pyramid / pooled vector
Task head or modality connector

Open the full decision guide

How it trains

Supervised labels, masked-image modeling, self-distillation, reconstruction, and image–text contrastive objectives produce different visual invariances and levels of spatial detail.

How inference runs

The encoder runs once per image or frame batch. Pooled outputs favor retrieval and classification; dense token maps preserve more localization detail for detection or multimodal reasoning.

Strengths

Reusable visual features
Efficient classification, retrieval, and perception
Can connect images to language models through a projector or cross-attention

Trade-offs

Resolution and patch size determine lost detail and token cost
Training objectives create different blind spots
A pooled vector is insufficient for many spatial tasks

Use it when

The system must understand or retrieve visual content
You can choose pooled versus spatial features based on the task
Small text, charts, and domain imagery are evaluated explicitly

Avoid or challenge it when

Visual generation is mistakenly expected from the encoder alone
Input resolution removes required details
A generic image benchmark substitutes for the actual domain

Illustrative published families

• Vision Transformer (ViT)
• Convolutional and hierarchical vision backbones
• The image tower in CLIP

Commonly combines with

Contrastive / dual encodersMultimodal fusionImage and video generators

Primary referencesVision Transformer CLIP

Modality interface · aligned spaces

Contrastive & dual encoders

#contrastive-dual-encoders

Encode two inputs independently—such as a query and document or image and caption—and train matching pairs to land near each other.

Mental model

Two readers meet in a shared coordinate system; they are fast because candidates do not interact until their vectors are compared.

Data flow

Paired inputs A and B
Separate or shared encoders
Projected normalized vectors
Similarity matrix
Contrastive matching loss

Open the full decision guide

How it trains

Batch contrastive objectives reward paired examples and treat other examples as negatives. Negative quality, duplicate semantics, and temperature materially affect the learned space.

How inference runs

Encode each side independently and compare vectors with dot product or cosine similarity. Precompute the large candidate side; optionally rerank top results with a cross-encoder.

Strengths

Scales retrieval to large corpora
Aligns modalities without a joint decoder
Enables zero-shot classification through label text in some settings

Trade-offs

Independent encoding misses fine-grained cross-input interactions
False negatives can distort training
Global similarity may ignore spatial, temporal, or compositional details

Use it when

Fast retrieval or matching is the first stage
One side can be indexed offline
Recall is followed by task-appropriate reranking when needed

Avoid or challenge it when

Every candidate needs deep token-to-token comparison
The task depends on precise spatial relationships
Raw similarity scores are assumed to be probabilities

Illustrative published families

• CLIP image–text dual encoder
• Dense passage retrieval
• Bi-encoder semantic search

Commonly combines with

Embedding modelsVision encodersRAGCross-encoder rerankers

Primary referencesCLIP Dense Passage Retrieval

Modality interface · understanding & generation

Multimodal fusion models

#multimodal-fusion

Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.

Mental model

“Multimodal” names the inputs and outputs, not one topology. Always ask where modalities are encoded, fused, and decoded.

Data flow

Text / image / audio / video
Modality encoders or tokenizers
Projector, cross-attention, or shared backbone
Joint representation / language decoder
Text, media, or action output

Open the full decision guide

How it trains

Systems combine contrastive alignment, captioning or next-token likelihood, masked objectives, paired instruction data, and sometimes separately pretrained frozen components.

How inference runs

Inputs are encoded into modality tokens or features, fused before or inside the language/generative backbone, and decoded into one or more modalities. Input understanding does not imply output generation.

Strengths

Grounds language in visual or acoustic inputs
Supports document, chart, image, video, and audio workflows
Can reuse strong pretrained modality components

Trade-offs

Modality imbalance and connector bottlenecks
Media tokens consume substantial context and compute
Evaluation must separate perception, grounding, reasoning, and generation

Use it when

The task genuinely depends on non-text evidence
Modality-specific slices and abstention are evaluated
The architecture exposes enough detail for cost and privacy review

Avoid or challenge it when

A transcript, OCR result, or structured extractor is sufficient
“Supports images” is assumed to mean precise perception
Sensitive media handling and retention are undefined

Illustrative published families

• Flamingo cross-attention architecture
• BLIP-2 learned querying connector
• Shared-token multimodal decoder systems

Commonly combines with

Vision / audio encodersDecoder-only language modelContrastive alignmentMedia generators

Primary referencesFlamingo BLIP-2

AI system architectures

Runtime structures around a model that add knowledge, tools, control, and verifiable behavior.

System · external knowledge

Retrieval-augmented & hybrid systems

#retrieval-augmented-hybrid-systems

Retrieve evidence at request time and provide it to a generator, classifier, or agent instead of relying only on model parameters.

Mental model

Open-book inference: retrieval chooses the evidence; generation uses it. Either half can fail independently.

Data flow

Ingest → chunk → index
User query
Dense, sparse, graph, or hybrid retrieval
Rerank + context assembly
Model answer with evidence links

Open the full decision guide

How it trains

Components may be trained separately or jointly: dual encoders learn retrieval, rerankers learn pairwise relevance, and generators learn grounded response behavior. Many practical RAG systems use pretrained components without end-to-end training.

How inference runs

A query may be rewritten or routed; multiple retrievers produce candidates; filters and a reranker select evidence; the model answers from a bounded context. Citation correctness requires checking that claims are actually supported.

Strengths

Fresh, private, or attributable knowledge without weight updates
Evidence can be inspected and access-controlled
Indexes and generators can evolve independently

Trade-offs

Bad chunking or retrieval creates confident answers from missing evidence
Ingestion freshness, permissions, and deletion become production systems
Retrieved content can carry prompt injection or poisoned data

Use it when

Answers depend on changing or proprietary sources
Users need traceable evidence
Recall, ranking, grounding, and answer quality can be evaluated separately

Avoid or challenge it when

The corpus is tiny enough for a deterministic lookup
No trustworthy source exists
Adding a vector database is being treated as a complete RAG design

Illustrative published families

• Original Retrieval-Augmented Generation research architecture
• Dense + sparse hybrid retrieval with reranking
• Graph or tool-based retrieval pipelines

Commonly combines with

Embedding modelsDual encodersRerankersText / chat modelsTool systems

Primary referencesRetrieval-Augmented Generation Dense Passage Retrieval

System · deliberate runtime control

Tool-using & reasoning systems

#tool-reasoning-systems

Place a model inside a controller that can plan, call typed tools, inspect results, revise, verify, stop, and hand control to a person.

Mental model

The model proposes; the runtime validates and executes. “Reasoning model” is a behavior/training/runtime description, not a single neural topology.

Data flow

Goal + policy + state
Model proposes answer or typed action
Schema / permission validation
Tool execution in a bounded environment
Observation → continue, verify, or stop

Open the full decision guide

How it trains

Tool demonstrations, supervised reasoning traces, outcome or process feedback, reinforcement learning, and verifiable tasks can shape behavior. Runtime scaffolding remains necessary even when the base model is post-trained for tool use.

How inference runs

The controller may spend multiple model calls and tool operations on one task. Budgets, deadlines, idempotency, approval gates, sandboxing, and explicit terminal conditions bound the loop.

Strengths

Access to current data, calculators, code, and enterprise systems
Can decompose, inspect, and verify multi-step work
Typed interfaces make capabilities and permissions explicit

Trade-offs

More latency, cost, and failure modes than one model call
Tool output and retrieved web content are untrusted inputs
Loops, repeated side effects, and excess authority create operational risk

Use it when

The task requires actions or information outside model weights
Intermediate results can be validated
Permissions, budgets, retries, and human approval are explicit

Avoid or challenge it when

One deterministic API call solves the task
The agent would receive broad credentials without containment
There is no evaluation for loops, tool errors, or unsafe actions

Illustrative published families

• ReAct-style reason-and-act loop
• Toolformer research approach
• Planner–executor and verifier patterns

Commonly combines with

Text / chat modelRAGCode sandboxPolicy engineHuman approval

Primary referencesReAct Toolformer

Architecture glossary

Short definitions for terms used across model papers and product documentation.

Attention: A learned weighted lookup that lets one position combine information from other positions.
Autoregressive: Factorizes an output into an ordered sequence and predicts each element from earlier elements.
Causal mask: Prevents a token from attending to future tokens during left-to-right language modeling.
Conditioning: Information—such as text, an image, a label, or control signal—that guides generation.
Cross-attention: Lets one sequence query the hidden states of another sequence or modality.
Denoising: Learning or applying a step that estimates clean data or a reverse update from a noisy sample.
Embedding: A learned numeric vector used as a representation for similarity or downstream prediction.
Expert / router: In MoE layers, an expert is a subnetwork and the router chooses which experts process each token.
Flow matching: Learns a time-dependent vector field that transports samples between probability distributions.
Grounding: Connecting a model response to supplied, inspectable evidence or an external source of truth.
KV cache: Stored attention keys and values from prior tokens, reused during autoregressive decoding.
Latent: A learned internal representation, commonly lower-dimensional than the original data.
Modality projector: A connector that maps one encoder’s features into the representation expected by another model.
Reranker: A second-stage model that scores a small candidate set with richer query–candidate interaction.
Recurrent state: A bounded hidden representation carried from one sequence step to the next.
Token / code: A discrete unit processed by a sequence model; it may represent text, an image patch, or compressed audio.

A capability is not an architecture

Customer-support copilot
= chat capability
+ decoder-only backbone
+ retrieval and reranking
+ typed tool controller
+ policy and human escalation

Product need

Test this baseline first

Why it fits

Watch closely

Open-ended assistant or coding copilot

Decoder-only LM + retrieval/tools as needed

Flexible generation and in-context task adaptation

Grounding, prompt injection, tool authority, serial latency

Classification, extraction, or reranking at volume

Encoder-only model or compact encoder–decoder

One-pass bounded prediction can be cheaper and easier to calibrate

Domain shift, input length, label calibration

Translation, transcription, or source-faithful transformation

Encoder–decoder model

Dedicated source memory conditions every generated token

Omissions, names/numbers, source length, decode latency

Semantic search over a large corpus

Dual encoder + vector index + optional reranker

Candidates can be embedded once and searched efficiently

Recall, negatives, filters, score calibration

Answers from current or private knowledge

Hybrid RAG + reranker + grounded generator

Knowledge remains external, inspectable, and refreshable

Ingestion, access control, missing evidence, poisoned context

Image creation or editing

Latent diffusion with U-Net or DiT denoiser

Compressed iterative generation supports strong conditioning and edits

Codec detail loss, sampling latency, rights and provenance

Video creation

Spatiotemporal latent diffusion or media-token generator

Both visual detail and temporal dependence are modeled

Identity drift, motion, duration, compute, audio sync

Speech recognition

Audio encoder–decoder designed for ASR

Separates acoustic understanding from text generation

Language, accent, overlap, noise, streaming delay

Long streaming sequence under a tight state budget

SSM/recurrent or attention–SSM hybrid

Carries compact recurrent state instead of a full attention cache

Exact long-range recall, kernel support, real throughput

Multimodal document or chart assistant

Vision encoder + language decoder + document retrieval

Preserves visual evidence while grounding answers in source pages

Small text, spatial references, media token cost, citations

Reliable actions in business systems

Tool-using LM inside a deterministic policy/controller

Language handles intent while code enforces permissions and side effects

Idempotency, approvals, retries, schemas, audit trail