Writing · AI evaluation

LLM Calibration: Brier Score and Expected Calibration Error Explained (with Python)

Robert Morong · 2026-10-23 · Drafted with AI assistance (Claude)

Accuracy tells you how often a model is right. Calibration tells you whether it knows when it is right. A calibrated model that says "90% sure" should be correct about 90% of the time on those answers. A model that says "95% sure" and is right 70% of the time is overconfident, and that is the failure that damages trust fastest, because users act on the confidence.

Where the confidence numbers come from

Calibration needs a confidence value per answer between 0 and 1. You can get it by asking the model to state a probability, from token probabilities where the API exposes them, or from agreement across repeated samples. Each has quirks, and stated confidences from chat models are often poorly calibrated, which is the thing you are measuring.

Brier score

The Brier score is the mean squared difference between the confidence and the outcome (1 if correct, 0 if wrong). Lower is better, and 0 is perfect.

def brier(conf, correct):
    return sum((c - y) ** 2 for c, y in zip(conf, correct)) / len(conf)

Expected calibration error

ECE groups answers into confidence bins, then compares the average confidence in each bin to the actual accuracy in it, weighted by how many answers fall in each bin.

def ece(conf, correct, bins=10):
    n = len(conf)
    total = 0.0
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        idx = [i for i, c in enumerate(conf)
               if (lo <= c < hi) or (b == bins - 1 and c == 1.0)]
        if not idx:
            continue
        acc = sum(correct[i] for i in idx) / len(idx)
        avg_conf = sum(conf[i] for i in idx) / len(idx)
        total += len(idx) / n * abs(acc - avg_conf)
    return total

A worked example

Take 100 answers where the model is right 70 times. Compare two models that are equally accurate:

correct = [1] * 70 + [0] * 30

conf_a = [0.95] * 100             # always says 95%
print(brier(conf_a, correct), ece(conf_a, correct))   # 0.2725  0.25

conf_b = [0.70] * 100             # always says 70%
print(brier(conf_b, correct), ece(conf_b, correct))   # 0.21    0.0

Both are 70% accurate. Model A is overconfident by 25 points, so its ECE is 0.25. Model B states exactly the accuracy it achieves, so its ECE is 0 and its Brier score is lower. In practice a model that says the same confidence every time is not very useful either, since it cannot tell easy cases from hard ones, so calibration should be read alongside accuracy and not instead of it.

How to read results

• Overconfidence (average confidence above accuracy) is the dangerous direction. Flag it explicitly.
• ECE depends on binning and gets noisy on small test sets, so with fewer than a couple of hundred answers treat it as a rough guide. Report the sample size with it.
• Brier score mixes accuracy and calibration together, so use ECE or a reliability plot to see why a score is bad.
• If calibration is poor you can often improve it after the fact by fitting a simple mapping from stated confidence to observed accuracy on held-out cases, but check it on cases the mapping was not fitted to.

This post covers the metrics, not a guarantee. A well-calibrated model on your test set can still be badly calibrated on inputs unlike it, so keep the test set representative. The wider picture of what to check before shipping is in the reliability checklist.

Keep reading

• The AI Reliability Checklist: 10 Checks Before You Ship an LLM Feature
• How to Tell If Your LLM Feature Is Actually Reliable (Before You Ship)
• Shuffle the answers and small models change their minds (ARC-Challenge audit)