Ai2 releases StateBench benchmark for long-context state tracking

Ai2, the Allen Institute for AI, released StateBench, a test set for measuring whether language models can follow changes to information about people, objects and variables across long documents. The benchmark has four difficulty levels, from values that never change to updates that depend on earlier updates, and presents the same problems in several formats. Ai2 published a technical report, data and code. In its evaluations, several open pretrained models and four frontier models handled easier levels but lost accuracy on harder ones as inputs grew; the longest general finite cases fell to 13% or lower. Models trained directly on the tasks showed recurrent and hybrid designs generalizing farther than transformers.