Writing · AI evaluation

Wilson Confidence Interval for LLM Accuracy (Python, No Dependencies)

Robert Morong · 2026-10-23 · Drafted with AI assistance (Claude)

If you report an eval result as "95% accurate" you have reported a number with no uncertainty attached. A confidence interval fixes that. The formula most people remember, the normal approximation, breaks in exactly the situation LLM evals often land in: small test sets and high accuracy. The Wilson score interval is the standard fix.

The problem with the textbook formula

The normal approximation says accuracy ± 1.96 × sqrt(p(1−p)/n). Try it on 19 correct out of 20:

import math
p, n = 19/20, 20
m = 1.96 * math.sqrt(p * (1 - p) / n)
print(p - m, p + m)   # 0.854..., 1.045...

The upper bound is 104.5%, which is not a possible accuracy. It also collapses to a zero-width interval if you get 20 out of 20, claiming certainty from 20 examples. Both failures come from using the same formula at the edges as in the middle.

The Wilson interval

import math

def wilson(k, n, z=1.96):
    """95% Wilson score interval for k successes out of n."""
    p = k / n
    denom = 1 + z**2 / n
    centre = p + z**2 / (2 * n)
    margin = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
    return (centre - margin) / denom, (centre + margin) / denom

It stays inside 0 to 1 and never collapses to zero width. Running it:

print(wilson(19, 20))    # (0.764, 0.991)
print(wilson(88, 100))   # (0.802, 0.930)
print(wilson(0, 20))     # (0.0, 0.161)

So 19 out of 20 is plausibly anywhere from 76% to 99%, and 88 out of 100 from 80% to 93%. Zero out of 20 does not mean "exactly 0%", it means "below about 16%".

How to use it in an eval

• Report accuracy as the point estimate plus the interval and the case count, for example "88% (95% CI 80% to 93%, n=100)".
• To compare against a ship threshold, use the lower bound. If your bar is 85% and the interval is 80% to 93%, you have not shown the feature clears the bar yet.
• If the interval straddles your threshold, collect more cases rather than rounding in your favour. The post on how many test cases you need has the numbers.

What it does not cover

Wilson is for a rate: pass/fail per case. For metrics like F1, AUC or RMSE use a bootstrap, which resamples the cases and recomputes the metric. It also assumes cases are independent. If you have several test cases that are paraphrases of each other, or the same question under different option orders, they are not independent, and the interval will look tighter than it should.

For comparing two systems on the same cases, a paired test is better than comparing two separate intervals, which is covered in the prompt regression testing post.

Keep reading

• How Many Test Cases Do You Need to Evaluate an LLM Feature?
• Prompt Regression Testing: How to Know a Prompt or Model Change Didn't Break Things
• Shuffle the answers and small models change their minds (ARC-Challenge audit)