StateBench: Testing how language models track state changes over long documents
A Hugging Face blog post about StateBench, which tests how language models track state changes over long documents.
What the article says
Ai2 introduces StateBench, a benchmark testing how language models track entities whose state changes across long documents (up to 64,000 tokens), in four difficulty levels from R-trivial to general finite problems. Pretrained models handle the easiest level, lose accuracy on aperiodic problems and stay at chance on general ones. Frontier models also drop sharply as inputs lengthen, falling to 13% or lower on the longest general problems. Gated DeltaNet and hybrid models trained directly on the task generalize far better than transformers, which never learned the hardest class. Published November 3, 2026 on Hugging Face.
Read the official article AnnuIA profile
Summary written by AnnuIA in its own words from the official page of Hugging Face. The content and rights belong to the publisher: please read the original article.