The number a benchmark exists to produce is a model’s score. This week’s StateBench release from Ai2 spends part of its own paper on a different number: the same model, on the same problems, near the ceiling when they are formatted like RULER retrieval prompts and near chance when they are formatted as spreadsheet edits. That result comes from the lab’s custom 275M hybrid and Gated DeltaNet checkpoints, which were not on the Hub when I looked, so the exact claim is not independently runnable. The dataset is public, though, and every instance carries an ID that joins it across all six renderings, so I ran the format comparison myself on public models.

I took the r-trivial level, the tier below state tracking: every assignment writes a variable’s first value, so the answer is just what each variable was first set to. Retrieval, no tracking, which makes it a clean probe, because a format gap there cannot be about reasoning. The ground truth is checkable without trusting the labels: the correct choice must equal each variable’s first executed assignment, stale-log distractor lines included in the prefixes and skipped, and all 4,770 checks across the pool passed. My sample is 76 problems: every 20th of the 1,590 instances that exist in all three short strata, in sorted ID order, with four dropped over a 3,000-token cap, tokens counted by the dataset’s own Olmo-3 tokenizer, the one its length strata are built on. The problems run 11 to 145 assignments, 2 to 16 variables, RULER renderings of 212 to 2,747 tokens, filler level matched across formats per instance. Five choices, so chance is 20%.

Same 76 instances, three public base models, CPU, log-likelihood scoring of the five choices. SmolLM2-135M: 85% as RULER (95% CI 76 to 92), 25% as spreadsheet edits, 76% as integer code. SmolLM2-360M: 97%, 42%, 93%. Qwen3-0.6B-Base: 100%, 88%, 100%. The smallest model’s spreadsheet interval covers the chance floor. Length-normalizing the scores changed no conclusion; it moved the 135M’s spreadsheet score from 25% to 22%.

The flips are the part worth keeping: 47 of the 76 problems the 135M solved as RULER it failed as spreadsheet edits, one went the other way. For the 360M, 43 of 76, again one backwards. For the Qwen, 9, none backwards.

Two details make the collapse specific rather than general. Integer code, the same numbers as bare a <- 98; lines plus commented-out stale-log distractors that look exactly like assignments, never collapses the way spreadsheet does: 76, 93, and 100%. And the RULER prompts are the longest of the three formats, median 1,024 tokens against 544 for the same traces as spreadsheet lines, so the easier format is the longer one. On the 24 filler-free instances the 360M missed 15 of 24 spreadsheet renderings and 1 of 24 RULER renderings of the same traces: the telegraphic “Enter 98 in cell B1.” syntax fails with no filler in the prompt at all, and whether the format’s native worksheet-note filler helps or hurts is not a clean signal in my run — it moves differently on the three models.

So the paper’s format effect is real and public-model sized, scaling with competence, chance floor at 135M, 42% at 360M, 88% at 0.6B. And at the r-trivial level what swings is not state tracking but the rendering’s fit to pretraining: a model that can recall the first thing it was told in one surface form and cannot in another is not varying in capability. StateBench earns its keep by shipping the same problem six ways and measuring its own format sensitivity; the caution is that its headline pretrained-model figure reports the max over those six, the optimistic edge of exactly this spread. Most benchmarks ship one rendering and call it the task. Before a gap between two models becomes a fact about the models, the format column has to exist, and StateBench is the rare benchmark that ships it.