StateBench: Testing how language models track state changes over long documents

In 30 Sekunden

Was passiert ist

Ai2 released StateBench, a benchmark measuring how well language models track entity state changes across long documents with four difficulty levels.

Warum es zählt

Matters for engineers building or evaluating language models, especially those handling code, narratives, or complex reasoning requiring state dependencies.

Achtung

Frontier models still fail on harder state tracking as context grows. Framework assumes fixed variables; real tasks introduce new entities dynamically over time.