Two endpoints serving GLM-5.3 Flash scored 22% and 29% on Humanity’s Last Exam this week. The reference deployment of the same weights, on the same questions, scores 55%. Same model, same questions, standardized sampling, no error anywhere in the transcript. The spread is out of Artificial Analysis’s new Endpoint Accuracy Index, which re-runs a fixed suite of tool calling, hard reasoning, and long-context recall against every provider serving the same open weights and reports each endpoint as a percentage of a self-hosted reference: 100 means the endpoint preserves what the model can do. Across the 100+ serverless endpoints it now covers, on six models, close to a quarter score below their reference, per the index’s launch coverage.

AA names serving configuration as the cause, and the sharpest is the output cap. Two GLM-5.3 Flash endpoints cap output near 32,000 tokens, and about half of their HLE-250 responses end before a final answer appears — which is how 22% and 29% happen against 55%. The cut books as a wrong answer, not an error. Two other endpoints produce roughly a third fewer reasoning tokens, with no truncation recorded, and score 48% on HLE-250 against the reference’s 55%. Two Muse Spark Open endpoints “return text tool arguments as numbers,” the same class wearing a parser’s clothes. And the usual suspect is mostly absent: five of six FP4-quantized GLM-5.3 endpoints are statistically indistinguishable from the FP8 reference, and all 22 DeepSeek V4.1 Flash endpoints score in line with their own reference — lower-precision weights do not explain most of the differences in this sample.

One methodology detail makes the cap worth singling out. AA’s harness requests “the maximum output tokens and context window supported by the endpoint” for every run, so each request already carries the endpoint’s own supported maximum — and on two endpoints, answers run into a ceiling near 32K that ends about half of the HLE-250 responses before a final answer appears (the benchmark repeats each question ten times, so that is a response-level rate; whichever way the cuts distribute across questions, each cut response scores as wrong). The GLM-5.3 Flash provider page lists 24 companies serving the same weights, 22 of them scored, composites running 79% to 104% of the reference: to a buyer working from the model card, one product in twenty-two wrappings.

When a cap does announce itself, the error text becomes the hazard. A Hermes Agent user’s OpenAI-compatible gateway started returning 400 max_tokens exceeds the limit of 4096 on requests that had no max_tokens configured. Hermes classified the message as a context overflow, parsed 4,096 as the model’s context window, and persisted it to a cache file. Every later agent then refused to start — a context window below the 64,000 minimum — for about eleven hours, across all of the user’s cron agents, until the cache row was deleted by hand, the failure message pointing at their config the whole time (issue). I reproduced the chain against upstream main: the wording walks past three guards, and all three are vocabulary lists. The classifier’s context-overflow patterns match “exceeds the limit”; both output-cap detectors miss the phrasing; the parse-side bail-out fires only on the literal words “output limit” or “output tokens,” so a generic “limit of N” pattern adopts 4,096 inside a plausibility band that accepts anything from 1,024 to 10 million. Requesting 200,000 drew a refusal naming 65,536 — the gateway’s cap as the gateway named it, not as anything verified it — and that number would have been cached too, as a window just above the floor that never trips the init check. But a wrong window does not only fail inits: Hermes derives its compression threshold from the resolved window, so an undersized 65,536 on a larger real one would compact conversations long before the true limit, silently, for as long as the row survives.

An output cap fails illegibly in both directions. Silently, it subtracts from what gets booked as model capability — AA’s index rests on that distinction, the same weights self-hosted being the 100, and the causes its results name are caps, parser quirks, and reasoning-length settings. Loudly, it hands the harness a number wrapped in error text, and a harness that learns limits from error wording files the number under whichever limit its patterns resemble, where it outlives the request that produced it. Hermes’s issue history already carries sibling reports of other gateways’ wordings walking past the same lists.

Two checks fall out. If you are choosing an endpoint, or re-scoring a model because it seemed to get dumber, set max_tokens to the endpoint’s documented maximum, send a short prompt — short enough that the context window cannot be the ceiling — that cannot finish within it, and read what comes back against what you asked for: a length finish with usage at the request means the answer hit the documented ceiling; a length finish with usage short of the request means a lower enforced ceiling. A rejection on asking for more hands you a number to check, though not always the right one — the Hermes gateway named 4,096 in one refusal and 65,536 in another, so treat the named cap as a claim to verify. A score that moves with the ceiling measured the pipe, not the model. And if you write a harness, a number that arrived inside an output-cap error should never reach a context-window cache — the Hermes issue shows the 400’s number outliving the request that produced it by eleven hours.