Model Architectures
Diffusion Transformers (DiT)
Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.
Mental model
DiT changes the denoiser backbone; it does not eliminate the diffusion process, latent codec, conditioning, or sampler.
Data flow
- Noisy latent
- Patchify + position / time conditioning
- Transformer blocks
- Predicted latent update
- Sampler step → repeat
How it trains
The model learns the same class of denoising, velocity, or flow objective as other diffusion systems while Transformer blocks mix information between latent patches.
How inference runs
A sampler repeatedly invokes the Transformer at different noise or flow times, then a decoder maps the final latent to media.
Strengths
- Transformer scaling and implementation ecosystem
- Global interactions between latent patches
- Natural path to shared multimodal token processing
Trade-offs
- Attention cost grows with latent token count
- Still requires repeated denoiser calls
- “DiT” alone does not specify codec, objective, conditioning, or sampler
Use it when
- Transformer scaling infrastructure is available
- Latent patch count and sampling latency fit the budget
- You need a flexible backbone for image or video generation
Avoid or challenge it when
- A small U-Net already meets the objective
- The label is being used as a complete architecture specification
- High-resolution token counts overwhelm the target runtime
Illustrative published families
- • DiT research family
- • Transformer denoisers inside latent image and video pipelines
Commonly combines with
Latent diffusionAutoencoder / VAEMultimodal conditioningMixture of Experts