Writing · AI evaluation

How to Tell If Your LLM Feature Is Actually Reliable (Before You Ship)

Robert Morong · 2026-10-22 · Drafted with AI assistance (Claude)

You wired up a model, tried a handful of prompts, the answers looked good, and you shipped. A week later it's doing something strange in production that you can't reproduce on your laptop. Sound familiar?

The root problem is simple: "it worked when I tried it" is an anecdote, not a measurement. Ten good-looking outputs tell you almost nothing about the next ten thousand. This post walks through the honest way to decide whether an LLM feature is reliable enough to ship — and where most teams fool themselves.

1. A demo is not production

The most expensive mistake in AI shipping is treating a successful demo as evidence of reliability. A demo is a sample of size ten, hand-picked, under ideal conditions. Production is a sample of size "everyone, forever," adversarial, at the edges.

Before you trust a feature, you need a number — and that number needs an uncertainty attached to it. Which brings us to the single most under-used idea in applied AI.

2. Every accuracy needs a confidence interval

Say you test on 20 examples and get 19 right: 95% accuracy. Ship it?

Run the math. With 19/20, the 95% confidence interval is roughly [76%, 99%]. In other words, your "95% accurate" feature could genuinely be 76% accurate — the difference between delightful and unusable — and your test couldn't tell. Twenty examples simply cannot support a precise claim.

The rule: never report an accuracy without its sample size and its interval. Use a Wilson interval for pass/fail rates (it behaves correctly near 0% and 100%, where the usual formula gives impossible answers), and a bootstrap for anything more complex like F1 or RMSE. If the interval is too wide to clear your bar, you don't have a reliability problem yet — you have a sample size problem, and the fix is more test cases, not more confidence.

3. Reliable means stable under trivial changes

Here's a test that catches a shocking number of "high-accuracy" features: take a multiple-choice or classification task, and reorder the options. Same question, same choices, different order.

If the chosen answer changes, the model was partly keying on position, not content, and your headline accuracy may be partly an artifact. Averaged over rotations the mean score can look unchanged while many individual answers flip, so count the flips, not just the score. The same goes for paraphrasing the prompt, changing case, or adding whitespace. A reliable system gives the same answer to the same question asked slightly differently. Measure the fraction of cases where the verdict survives these meaning-preserving perturbations; that robustness rate is often far below the raw accuracy, and it's closer to what users will actually experience.

4. Reliable means stable run to run

Run the exact same inputs five times. Do you get the same verdicts every time? Sampling and non-determinism can flip individual answers even while the average accuracy looks steady. For anything user-facing, a high flip rate is a defect in itself — the same user asking the same question twice should not get contradictory answers.

5. Confidence should match correctness

The failure mode that burns trust fastest is overconfidence: the model says "definitely" and is wrong. Check calibration — does a stated 90% confidence actually correspond to ~90% correctness? Expected Calibration Error and the Brier score make this concrete. A feature that is accurate and knows when it's unsure is far safer to ship than one that is slightly more accurate but confidently wrong.

6. The verdict: Ship / Caution / Not-ready

Put it together into one honest call:

• Ship — accuracy interval clears your bar, robust to perturbation, consistent across runs, well-calibrated.
• Caution — usable with named caveats: which metric is weak, at what sample size, which perturbation breaks it, and exactly what would move it to Ship.
• Not-ready — a load-bearing check fails: the accuracy interval dips below threshold, robustness collapses, or the model is confidently wrong.

An honest Caution is worth more than a flattering Ship. The point isn't to make the feature look good; it's to know what you're actually shipping.

Keep reading

• Your LLM Benchmark Score Is Probably Lying to You
• The AI Reliability Checklist: 10 Checks Before You Ship an LLM Feature
• Shuffle the answers and small models change their minds (ARC-Challenge audit)