Chasing the Batch-1 Floor: Ling-3.0-flash Speculative Decode on Blackwell

За 30 секунд

Что произошло

LMSYS optimized Ling-3.0-flash speculative decoding on Blackwell GPUs, achieving 2.1x single-request throughput and 54% lower latency through host pipelining and kernel fusion.

Почему это важно

Matters for engineers building low-latency inference systems where batch-1 decode performance directly impacts user-facing latency and throughput at scale.

На что обратить внимание

Results are specific to this model architecture, workload, and hardware setup; accept length of 9.95 depends on prompt and output distribution, not generalizable across all use cases.