Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating

In 30 Sekunden

Was passiert ist

Researchers propose RMM, a fixed-lag smoothing approach to decide which cached tokens to keep in language models with bounded memory.

Warum es zählt

Matters for engineers optimizing inference on resource-constrained systems where KV cache management directly impacts throughput and latency.

Achtung

Method matches existing approaches on standard benchmarks; gains appear only when token reuse is sharp and endogenous, not typical in real workloads.