Architectures de modèles
Autoencoders, VAEs & learned tokenizers
Learn an encoder that compresses data and a decoder that reconstructs it; variational and quantized variants shape the latent space for sampling or downstream generation.
Modèle mental
A learned codec. It decides which information survives before another model generates in the compressed space.
Flux de données
- Image, audio, or other signal
- Encoder
- Continuous latent or discrete codes
- Optional latent generator
- Decoder reconstruction
Entraînement
A reconstruction objective preserves perceptual content. VAEs add a distribution regularizer; vector-quantized models map representations to a discrete codebook and add codebook/commitment objectives.
Exécution de l’inférence
Encode for compression or editing; sample or generate a latent/code sequence; decode to the original modality. Reconstruction quality limits the best possible downstream output.
Atouts
- Reduces the dimensionality of expensive media
- Creates reusable continuous latents or discrete tokens
- Separates representation learning from the generative prior
Compromis
- Compression discards detail
- Latent geometry may not match downstream semantics
- Quantized codebooks can be underused or introduce artifacts
À utiliser lorsque
- Raw media is too expensive to model directly
- A generative model needs a compact continuous or discrete space
- Reconstruction failures are measured by content slice
À éviter ou remettre en question lorsque
- Lossless reconstruction is mandatory
- The codec domain differs materially from production data
- Decoder artifacts would be mistaken for generator failures
Familles publiées à titre d’exemple
- • Variational Autoencoder (VAE)
- • VQ-VAE discrete visual/audio tokenizers
- • Autoencoders used by latent diffusion
Souvent combinée avec
Latent diffusionAutoregressive media modelsDiffusion Transformer