モデルアーキテクチャ
Autoregressive visual & audio models
Convert images, video, or audio into discrete codes and predict those codes in an ordering, often with a causal Transformer.
理解のためのモデル
Make media look like a language: tokenize it, choose an ordering, predict the next code, then decode.
データフロー
- Media tokenizer / codec
- Ordered discrete codes
- Causal token model
- Generated code sequence
- Media decoder
学習方法
A learned codec is trained for reconstruction; the prior minimizes next-code cross-entropy, optionally conditioned on text, prior frames, semantic tokens, or coarse-scale codes.
推論の実行方法
Codes are generated serially or in hierarchical/blockwise schedules, then decoded. Ordering determines latency and which dependencies are easy to model.
強み
- Unified token-based modeling across modalities
- Compatible with causal language-model infrastructure
- Exact discrete likelihood for the chosen tokenization
トレードオフ
- Long code sequences create serial latency
- Tokenizer artifacts bound output quality
- A one-dimensional ordering can be awkward for spatial or multiscale structure
適する場合
- A strong discrete codec exists
- Cross-modal token modeling or continuation is important
- Hierarchical generation can meet latency goals
避ける・再検討する場合
- Long serial decoding misses the budget
- The codec loses critical perceptual detail
- A continuous diffusion representation is substantially simpler
公開された方式の例
- • ImageGPT-style pixel/token modeling
- • VQ-token image generators
- • AudioLM-style codec-token hierarchy
よく組み合わせる要素
Decoder-only TransformerVQ-VAE / neural codecMultimodal token fusion