Parallel All the Way Down: Beyond Single-Token Generation with Speculative Decoding

За 30 секунд

Что произошло

vLLM now supports three parallel drafting algorithms, P-EAGLE, DFlash, and DSpark, that generate multiple candidate tokens simultaneously instead of sequentially.

Почему это важно

Engineers optimizing LLM inference should care when speculative decoding bottlenecks limit throughput or when tuning speculation length becomes operationally burdensome.

На что обратить внимание

Performance varies significantly across models, tasks, and hardware. Figure 1 plots were corrected after initial publication due to environment setup errors affecting absolute numbers.