モデルアーキテクチャ
Audio, speech & music models
A family of task-specific pipelines: speech recognition, speech synthesis, audio understanding, voice conversion, sound generation, and music generation are not one architecture.
理解のためのモデル
Choose the task first. ASR maps audio to text; TTS and music generation map conditions to acoustic or codec representations; understanding maps audio to labels or embeddings.
データフロー
- Waveform or spectrogram
- Acoustic / codec representation
- Encoder, seq2seq, diffusion, or token generator
- Text, labels, codec tokens, or waveform
- Task-specific decoding
学習方法
Objectives vary: transcription uses sequence likelihood, representation learning may mask audio, codec generators predict discrete audio tokens, and diffusion systems learn denoising in waveform, spectrogram, or latent space.
推論の実行方法
ASR usually encodes audio then decodes text; generation may predict codec tokens autoregressively or iteratively denoise. Streaming constraints can require chunked encoders and bounded look-ahead.
強み
- Speech transcription and translation
- Natural speech, sound, and music synthesis
- Audio embeddings support search, moderation, and classification
トレードオフ
- Accents, noise, overlap, code-switching, and domain vocabulary shift quality
- Long-form generation needs temporal structure and consistency
- Voice identity, consent, watermarking, and copyright need explicit governance
適する場合
- The modality itself carries needed information
- You have task-, language-, and environment-specific evaluations
- Latency and streaming requirements are designed up front
避ける・再検討する場合
- A transcript already contains all relevant information
- Identity cloning lacks informed consent
- One benchmark is being used as proof across languages and acoustic conditions
公開された方式の例
- • Whisper-style encoder-decoder ASR
- • AudioLM-style hierarchical codec-token generation
- • Waveform or latent diffusion audio systems