Model Architectures
Model architecture field guide
Generative Model Architectures
Understand what actually sits behind text, embedding, image, video, audio, and multimodal products, then choose a backbone, representation, generator, and system wrapper deliberately.
22 architecture families · 5 groups · one page per family
The essential distinction
A capability is not an architecture
“Chat,” “image,” and “multimodal” describe an interface. Decoder-only, state-space, DiT, and MoE describe internal computation. RAG and tool use describe the runtime system around a model. One production system can legitimately carry all three kinds of label.
= chat capability
+ decoder-only backbone
+ retrieval and reranking
+ typed tool controller
+ policy and human escalation
Write an architecture passport
- 1Capability: Inputs, outputs, user job, and required evidence
- 2Representation: Tokens, vectors, pixels, codec codes, or continuous latents
- 3Backbone: Encoder, decoder, seq2seq, attention, SSM, or hybrid
- 4Capacity: Dense or expert-routed; total versus active parameters
- 5Generation: Autoregressive, diffusion/flow, adversarial, or deterministic
- 6System: Retrieval, tools, validators, permissions, monitoring, and humans
Browse all 22 families
Each family has its own page with training, inference, strengths, trade-offs, and selection guidance.
Product capabilities
What the product accepts and returns. These are deployment shapes, not mutually exclusive neural architectures.
Text & chat language models
Generate text one token at a time, usually with a causal decoder trained for next-token prediction and then adapted to follow instructions.
Embedding models
Map text, images, audio, users, or items into vectors whose geometry is trained to preserve a useful notion of similarity.
Image generation & editing
Synthesize or transform pixels, most often through a text or image conditioner, a latent generator, and an image decoder.
Video generation
Generate spatial and temporal structure together using frame, patch, or latent video representations with cross-frame computation.
Audio, speech & music models
A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.
Core backbones
How information moves through the trainable network. A deployed model may combine several of these choices.
Encoder-only Transformers
Let every input token attend to tokens on both sides, producing contextual representations rather than an open-ended generation loop.
Decoder-only Transformers
Use a causal mask so each position sees only earlier tokens, matching the left-to-right generation process used by most general-purpose chat models.
Encoder–decoder Transformers
Encode an input bidirectionally, then generate an output causally while cross-attending to the encoded source.
Mixture of Experts (MoE)
Replace selected dense sublayers with many expert networks and a learned router that activates only a small subset for each token.
State-space & recurrent models
Update a compact state as tokens arrive instead of retaining an explicit attention relationship between every pair of positions.
Representation & generation
How images, audio, video, and other high-dimensional outputs are represented, learned, and sampled.
Autoencoders, VAEs & learned tokenizers
Learn an encoder that compresses data and a decoder that reconstructs it; variational and quantized variants shape the latent space for sampling or downstream generation.
Generative Adversarial Networks (GANs)
Train a generator to fool a discriminator while the discriminator learns to distinguish generated samples from training data.
Normalizing flows
Transform a simple distribution into a complex data distribution through a sequence of invertible mappings with tractable Jacobians.
Diffusion & score-based generation
Learn to reverse a gradual noising process, producing data by repeatedly transforming noise toward a sample.
Latent diffusion
Run diffusion in an autoencoder’s lower-dimensional latent space, then decode the generated representation back to pixels or another signal.
Diffusion Transformers (DiT)
Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.
Autoregressive visual & audio models
Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.
Modality interfaces
How vision, text, audio, and other modalities are encoded, aligned, fused, or connected to a generator.
Vision encoders
Transform pixels into feature maps, patch tokens, or pooled vectors that downstream classifiers, retrievers, decoders, or multimodal models can consume.
Contrastive & dual encoders
Encode two inputs independently (such as a query and document or image and caption) and train matching pairs to land near each other.
Multimodal fusion models
Connect modality-specific encoders and generators through projection, cross-attention, shared token spaces, or combinations of these mechanisms.
AI system architectures
Runtime structures around a model that add knowledge, tools, control, and verifiable behavior.
Retrieval-augmented & hybrid systems
Retrieve evidence at request time and provide it to a generator, classifier, or agent instead of relying only on model parameters.
Tool-using & reasoning systems
Place a model inside a controller that can plan, call typed tools, inspect results, revise, verify, stop, and hand control to a person.