Architectures de modèles
Diffusion Transformers (DiT)
Use a Transformer over noisy image or video patches as the denoising network inside a diffusion or flow-based generative process.
Modèle mental
DiT changes the denoiser backbone; it does not eliminate the diffusion process, latent codec, conditioning, or sampler.
Flux de données
- Noisy latent
- Patchify + position / time conditioning
- Transformer blocks
- Predicted latent update
- Sampler step → repeat
Entraînement
The model learns the same class of denoising, velocity, or flow objective as other diffusion systems while Transformer blocks mix information between latent patches.
Exécution de l’inférence
A sampler repeatedly invokes the Transformer at different noise or flow times, then a decoder maps the final latent to media.
Atouts
- Transformer scaling and implementation ecosystem
- Global interactions between latent patches
- Natural path to shared multimodal token processing
Compromis
- Attention cost grows with latent token count
- Still requires repeated denoiser calls
- “DiT” alone does not specify codec, objective, conditioning, or sampler
À utiliser lorsque
- Transformer scaling infrastructure is available
- Latent patch count and sampling latency fit the budget
- You need a flexible backbone for image or video generation
À éviter ou remettre en question lorsque
- A small U-Net already meets the objective
- The label is being used as a complete architecture specification
- High-resolution token counts overwhelm the target runtime
Familles publiées à titre d’exemple
- • DiT research family
- • Transformer denoisers inside latent image and video pipelines
Souvent combinée avec
Latent diffusionAutoencoder / VAEMultimodal conditioningMixture of Experts