AI evaluation · reproducible audit

Shuffle the answers and small models change their minds

I scored four small open models on all 1,165 four-option questions in ARC-Challenge, each shown four times with only the order of the answer options changed. Two of the three models that carry real signal changed their chosen answer on 60% or more of the questions, and model size did not explain the differences.

Robert Morong · October 22, 2026 · code and raw predictions

The short version

• A single score hides the problem. Accuracy per ordering was nearly identical for every model, because cycling the options puts the right answer in each slot equally often and the bias cancels in the average. The bias shows up in two other numbers: how often the chosen answer changes (the flip rate) and accuracy broken out by where the correct answer sits.
• Within one family, the effect shrinks with size. Qwen2.5-0.5B changed its answer on 74.6% of questions. Qwen2.5-1.5B changed on 31.6%.
• Across families, size told me nothing. SmolLM2-1.7B is about the same size as Qwen2.5-1.5B and changed its answer on 59.8% of questions, almost twice as often.

What I ran

ARC-Challenge test split, every question that has exactly four options (1,165 of 1,172). Each question is shown four times with the options rotated cyclically (A-B-C-D, then B-C-D-A, and so on), so every option appears in every slot once. The model is scored on its next-token probabilities for the letters A to D after a plain completion prompt, with no chat template. That makes 4,660 predictions per model. Model revisions, the dataset revision and the random seed are pinned, and every number below is produced by code from the raw per-prediction files, not typed by hand.

Results

The 95% bootstrap intervals over questions for the flip rates are 72.1 to 77.2 (Qwen 0.5B), 28.8 to 34.2 (Qwen 1.5B) and 56.9 to 62.6 (SmolLM2-1.7B). The full table, with every interval, is in RESULTS.md.

What the numbers say

Qwen2.5-0.5B has a strong slot bias

It is 26.0% accurate when the correct answer sits in slot D, which is about chance for four options, and 60.3% when it sits in slot B. It also picks D only 11.9% of the time. The difference across slots is not noise (chi-square p below 1e-70).

The bias shrinks with size inside one family, but does not vanish

Qwen2.5-1.5B flips on 31.6% of questions instead of 74.6%. Its slot dependence is much smaller but still detectable: 69.2% accuracy when the answer is in slot A versus 79.1% in slot C.

SmolLM2-1.7B is unstable without a simple slot preference

Its picks are spread almost evenly over the four letters (26.8, 22.9, 23.2 and 27.0%) and its accuracy by correct-answer slot is flat (p = 0.19). Yet 59.8% of its answers still change when only the order changes. So "this model likes letter C" is not the whole story of order sensitivity, and checking the pick distribution alone would have missed it.

The flips are not pure noise

A model that ignored content and chose slots according to its own observed slot preferences would flip on about 98.5% of questions. Accuracy on the questions that do flip is 39.8%, 44.7% and 41.1% for the three informative models, above the 25% chance floor. The questions that never flip are more accurate (75.0%, 88.3%, 83.5%), but that group is selected for being stable, so it should not be read as the model's ability on easy items.

SmolLM2-360M tells us nothing, and I am saying so

It scores at chance and picks slot C 85% of the time. Its 100% flip rate is a floor effect, not evidence about robustness. It is in the table for completeness and should not be used in comparisons.

Why this matters if you run evaluations

If you report one accuracy number from one option ordering, you may be reporting a mix of what the model knows and where the answer happened to be placed. Two cheap habits help: rotate or shuffle the options and report the flip rate next to accuracy, and break accuracy out by the slot of the correct answer. A large spread across slots, or a high flip rate at similar accuracy, is a reliability problem that a single score cannot show.

Limits you should hold me to

• Four small models from two families, one prompt template, label-probability scoring rather than free-text generation. This says nothing about frontier models, and four models are not a scaling law.
• Four cyclic rotations, not all 24 permutations, and option order only: no paraphrase or other perturbations.
• Precision differs across models. The 0.5B and 360M models ran in float32. The 1.5B and 1.7B models ran in bfloat16 because float32 ran out of memory on the 8 GB CPU machine I used. bfloat16 produced exact probability ties (60 for Qwen 1.5B, 121 for SmolLM2-1.7B), which the code breaks by slot order. A control that re-decides every tie at random (200 reseeds) moved accuracy by at most 0.3 point and the flip rates from 31.6% to 32.1% and from 59.8% to 60.3%, so ties do not drive the results. Precision is still confounded with size in this design, so size comparisons carry that caveat.
• Runs were executed in resumable chunks after container restarts. Each question is scored independently with fixed weights, so resuming does not change any prediction.
• Not peer reviewed. Different hardware can change the last digits. Analysis code and drafting were AI-assisted; the headline numbers were independently recomputed with a second code path and match.

Check it yourself

Everything is in the repository: pinned revisions, the scripts, the raw predictions for all four models, SHA-256 hashes of each results file, and the commands to rerun it (about 40 to 50 minutes per model on two CPU cores). If you rerun it and get different numbers, or find a flaw, that is a useful outcome and I want to know.

Code, data and provenance on GitHub