Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating

За 30 секунд

Что произошло

Researchers propose RMM, a fixed-lag smoothing approach to decide which cached tokens to keep in language models with bounded memory.

Почему это важно

Matters for engineers optimizing inference on resource-constrained systems where KV cache management directly impacts throughput and latency.

На что обратить внимание

Method matches existing approaches on standard benchmarks; gains appear only when token reuse is sharp and endogenous, not typical in real workloads.