Writing · AI evaluation

Prompt Regression Testing: How to Know a Prompt or Model Change Didn't Break Things

Robert Morong · 2026-10-23 · Drafted with AI assistance (Claude)

You tweak a prompt, swap a model version, or change a system message. Accuracy on your test set goes from 80% to 84%. Did it actually improve, or did you just reshuffle which cases you get right? The way to find out is to compare the two versions case by case instead of comparing two headline numbers.

Step 1: run both versions on the same frozen cases

Keep a fixed test set, and store the old version's per-case results (correct or not) as a baseline. When you change something, re-run on the exact same cases. Without per-case results you can only compare totals, and totals hide what happened.

Step 2: count what was fixed and what broke

For each case there are four outcomes: right before and after, wrong before and after, fixed (wrong then right), and broke (right then wrong). Only the last two carry information about the change.

Two scenarios with the same kind of headline:

• Old prompt 80 correct, new prompt 84 correct, but it fixed 12 cases and broke 8. The net gain is 4.
• Old prompt 80 correct, new prompt 80 correct, but it fixed 10 and broke 10. Accuracy is flat, yet 20 cases changed behaviour, which matters if those cases matter to your users.

Step 3: use an exact paired test

If the change had no effect, then among the cases that changed, "fixed" and "broke" should be roughly a coin flip. An exact sign test checks that, using only the cases that changed:

from math import comb

def paired_sign_test(old, new):
    """old, new: lists of 0/1 (1 = correct) on the SAME test cases.
    Returns (fixed, broke, two-sided exact p-value)."""
    fixed = sum(1 for o, n in zip(old, new) if o == 0 and n == 1)
    broke = sum(1 for o, n in zip(old, new) if o == 1 and n == 0)
    m = fixed + broke
    if m == 0:
        return fixed, broke, 1.0
    k = min(fixed, broke)
    p = 2 * sum(comb(m, i) for i in range(k + 1)) / 2**m
    return fixed, broke, min(1.0, p)

Running it on the examples above (100 cases each):

• 80 to 84 correct, 12 fixed, 8 broke: p = 0.50. That 4-point improvement is well within noise. You cannot say the new prompt is better.
• 80 to 80 correct, 10 fixed, 10 broke: p = 1.0. No evidence of a net change, but a lot of churn.
• 80 to 90 correct, 14 fixed, 4 broke: p = 0.031. This one is reasonably convincing: a 10-point gain driven by many more fixes than breaks.

The same four-point gain that looked like a win in the first case is indistinguishable from chance on 100 cases, which is why test set size matters so much for comparisons.

Make it a habit

• Freeze the test set and the baseline per-case results in version control.
• On every prompt or model change, report fixed, broke, and the p-value, not just accuracy.
• Read the "broke" cases by hand. A change that fixes easy cases and breaks important ones can look fine in aggregate.
• Pick a rule for failing the build, for example "any change that breaks more than N cases, or that drops accuracy below the ship threshold, needs a human sign-off".
• If you run the same case multiple times, remember that model output can vary between runs, so check run-to-run consistency first or you will be comparing noise to noise.

This tests one thing: whether behaviour on your cases changed. It says nothing about cases you did not include, so keep adding the failures you discover in production to the test set.

Keep reading

• How Many Test Cases Do You Need to Evaluate an LLM Feature?
• Wilson Confidence Interval for LLM Accuracy (Python, No Dependencies)
• Shuffle the answers and small models change their minds (ARC-Challenge audit)