Writing · AI evaluation

Your LLM Benchmark Score Is Probably Lying to You

Robert Morong · 2026-10-22 · Drafted with AI assistance (Claude)

"Our model hits 88% on the benchmark." Great — but 88% of what, measured how, on how many examples, and how much would it move if you changed one trivial thing? Most published and internal LLM benchmark numbers quietly fail at least one of these questions. Here are the three ways a benchmark lies, and how to tell.

Lie #1: No confidence interval

A benchmark score is an estimate from a finite sample, so it has uncertainty — yet most are reported as a bare number. On 200 examples, an 88% has a 95% confidence interval of roughly ±4.5 points. So an "88% vs 85%" comparison between two models, on 200 examples, is not a real difference — it's inside the noise. Teams routinely pick models, write launch posts, and set roadmaps on gaps that vanish under a confidence interval.

The fix: every benchmark number gets a sample size and an interval (Wilson for accuracy, bootstrap for F1/AUC). And before you call model B better than model A, check that the interval of the difference excludes zero — ideally with a paired test on the same items.

Lie #2: Option-order bias

This one is cheap to check and easy to miss. Take a multiple-choice benchmark and shuffle the answer options — same questions, same choices, different order. How much the average score moves depends on how you shuffle, so measure it instead of assuming. In my own audit of four small open models on ARC-Challenge, the mean score barely changed across four cyclic rotations, yet the models changed their chosen answer on 32% to 75% of questions. Both numbers matter, and a single score shows neither.

Why? The model is partially keying on the position of the answer (say, a mild preference for "C"), not purely on its content. That means the benchmark is measuring a mix of real capability and a positional artifact. If your headline score drops when you shuffle options, the real capability number is lower than advertised — and a competitor who happened to order their options differently isn't actually behind you.

Always report accuracy averaged over randomized option orders, and report how much the shuffle moved the score. The size of that movement is itself a reliability metric.

Lie #3: The test set is too small (or contaminated)

Two quieter problems:

• Too small. A benchmark of 50 items can't distinguish 90% from 80% with any confidence (the 95% interval is roughly ±8 points at 90% and ±11 at 80%). If the test set can't support the precision implied by the headline, the headline is marketing, not measurement.
• Contaminated. If the model (or the person reporting) has seen the test items — because they're on the public web, or were used to tune prompts — the score is inflated and tells you nothing about new inputs. Note the contamination risk explicitly; a benchmark you tuned on is a benchmark you can't trust.

What a trustworthy benchmark looks like

• Every score carries its sample size and confidence interval.
• Multiple-choice scores are averaged over randomized option orders, with the shuffle sensitivity reported.
• Differences between systems are only called real when a paired test and the interval of the difference support them.
• The test set is large enough for the claimed precision and uncontaminated, with provenance recorded (data version, seed, environment).
• Results are reproducible from the stated inputs — run it twice, get the same number, and say so.

None of this is exotic. It's just the difference between a number you can bet on and a number that looks good in a slide.

Keep reading

• How to Tell If Your LLM Feature Is Actually Reliable (Before You Ship)
• The AI Reliability Checklist: 10 Checks Before You Ship an LLM Feature
• Shuffle the answers and small models change their minds (ARC-Challenge audit)