Autoregressive visual & audio models

Как проходит обучение

A learned codec is trained for reconstruction; the prior minimizes next-code cross-entropy, optionally conditioned on text, prior frames, semantic tokens, or coarse-scale codes.

Как выполняется инференс

Codes are generated serially or in hierarchical/blockwise schedules, then decoded. Ordering determines latency and which dependencies are easy to model.

Сильные стороны

• •Unified token-based modeling across modalities
• •Compatible with causal language-model infrastructure
• •Exact discrete likelihood for the chosen tokenization

Компромиссы

• •Long code sequences create serial latency
• •Tokenizer artifacts bound output quality
• •A one-dimensional ordering can be awkward for spatial or multiscale structure

Использовать, когда

• A strong discrete codec exists
• Cross-modal token modeling or continuation is important
• Hierarchical generation can meet latency goals

Избегать или пересмотреть, когда

• Long serial decoding misses the budget
• The codec loses critical perceptual detail
• A continuous diffusion representation is substantially simpler