StateBench: Testing how language models track state changes over long documents

Why it matters

StateBench is a new benchmark designed to measure how well language models track changing information about entities across long documents, a skill underlying tasks like following code variable reassignments or narrative events such as John handing Mary an item that Edward had earlier placed in John's backpack. The benchmark organizes state tracking into four difficulty levels, ranging from immutable assignments, through reassignment by value and self-referential updates, up to assignment by reference, where one variable takes another's current value. These levels are grounded in Krohn–Rhodes theory, which decomposes state-tracking problems into atomic primitives. The work was motivated by known theoretical limits of transformers on hard state tracking, which have driven interest in recurrent and hybrid designs like Gated DeltaNet. Evaluations of open pretrained models showed most handled the easiest level well, degraded on mid-level problems as inputs lengthened, and performed at chance on the hardest level, broadly matching theoretical predictions about where each architecture should fail.