Writing

Notes on measuring AI reliably

Independent research and practical guides on evaluating language models without fooling yourself: confidence intervals, robustness, consistency and calibration.

• Shuffle the answers and small models change their minds
  Primary research. Four small open models, all 1,165 four-option ARC-Challenge questions, four option orders. Code and raw predictions are public.
• How to Tell If Your LLM Feature Is Actually Reliable (Before You Ship)
  A demo passing is not a feature working. Here's the honest, measurable way to decide whether an LLM feature is production-ready — accuracy with confidence intervals, robustness, consistency, and calibration.
• Your LLM Benchmark Score Is Probably Lying to You
  Option-order bias, no confidence intervals, and tiny test sets make most LLM benchmark numbers meaningless. Here's how to spot a benchmark you can't trust — and how to run one you can.
• The AI Reliability Checklist: 10 Checks Before You Ship an LLM Feature
  A practical, copy-pasteable checklist for deciding whether an LLM feature is production-ready: accuracy with intervals, robustness, consistency, calibration, and reproducibility.
• How Many Test Cases Do You Need to Evaluate an LLM Feature?
  A sample-size table for LLM evals: how wide your accuracy confidence interval is at 20, 50, 100, 500 and 1,000 test cases, and how many cases you need for a given precision.
• Wilson Confidence Interval for LLM Accuracy (Python, No Dependencies)
  Why the usual plus-or-minus formula fails for LLM eval accuracy near 0% and 100%, and a 6-line Python Wilson score interval you can paste into any eval script.
• Prompt Regression Testing: How to Know a Prompt or Model Change Didn't Break Things
  How to compare an old and new prompt or model on the same test cases, count what got fixed and what broke, and use an exact paired test instead of eyeballing two accuracy numbers.
• LLM Calibration: Brier Score and Expected Calibration Error Explained (with Python)
  What it means for an LLM's stated confidence to be calibrated, how to compute the Brier score and expected calibration error (ECE) in plain Python, and how to read the results.