Writing · AI evaluation
How Many Test Cases Do You Need to Evaluate an LLM Feature?
The honest answer is "enough that the confidence interval on your accuracy is narrower than the decision you are trying to make." That sounds abstract, so here are the actual numbers. Everything below is a 95% Wilson interval, computed with the short function in the Wilson interval post.
What a test set of a given size can tell you
Suppose your feature gets 80% of the test cases right. How uncertain is that number?
- With 20 cases (16 right), the interval is roughly 58% to 92%. You cannot tell a mediocre feature from a good one.
- With 50 cases (40 right), roughly 67% to 89%.
- With 100 cases (80 right), roughly 71% to 87%.
- With 200 cases (160 right), roughly 74% to 85%.
- With 500 cases (400 right), roughly 76% to 83%.
- With 1,000 cases (800 right), roughly 77% to 82%.
The interval shrinks with the square root of the sample size, so each extra bit of precision costs more than the last. Going from 100 to 200 cases helps a lot. Going from 500 to 1,000 helps much less.
How many cases for a target precision
If you decide in advance how wide an interval you can live with, you can work out the test set size. These are the case counts needed for the 95% Wilson interval to have a half-width at or under the target, at different true accuracy levels:
- At 70% accuracy: about 78 cases for ±10 points, 320 for ±5, 894 for ±3, 2,014 for ±2.
- At 80% accuracy: about 60 cases for ±10 points, 245 for ±5, 681 for ±3, 1,535 for ±2.
- At 90% accuracy: about 34 cases for ±10 points, 141 for ±5, 384 for ±3, 864 for ±2.
- At 95% accuracy: about 24 cases for ±10 points, 82 for ±5, 208 for ±3, 465 for ±2.
Two things stand out. Features that are already very accurate need far fewer cases to pin down, and being precise to within two points is expensive at any accuracy level.
Comparing two prompts or models needs more than you think
Everything above is about measuring one system. Comparing two is harder, because you are trying to detect a difference, and differences are small. If a new prompt moves accuracy from 80% to 84%, a 100-case test set cannot reliably confirm that, because each of the two intervals is about ±8 points wide.
Two things help. First, run both systems on the same cases and use a paired test, which looks only at the cases where the two systems disagree. That removes the noise from cases both get right or both get wrong, and is covered in the prompt regression testing post. Second, decide the smallest difference you would act on before you look at results, and size the test set for that.
Practical rules
- Report the sample size next to every accuracy number, always.
- Fewer than about 50 cases is a smoke test, not a measurement.
- Around 200 cases gets you to roughly ±4 to ±6 points at typical accuracy levels, which is often enough for a go or no-go call.
- If the interval is too wide to clear your ship threshold either way, the answer is "collect more cases", not "ship it" or "don't".
- Cases must be representative. A thousand easy, near-duplicate cases give you a narrow interval around the wrong number.
The intervals assume cases are independent and that they were not used to tune the prompt. Cases drawn from the same source with near-duplicates will make the true uncertainty larger than the interval suggests.
Run these checks without hand-rolling them
The AI Reliability Test Kit is a Python harness that measures accuracy with Wilson and bootstrap confidence intervals, robustness to rewording and option reordering, run-to-run consistency and calibration, then writes a shareable HTML scorecard with a Ship / Caution / Not-ready verdict. It runs offline with a built-in mock model, or against OpenAI, Anthropic and any OpenAI-compatible or local endpoint.
See what is in the Test Kit Try the free option-order check first