Reliability Scorecard

Support ticket classifier (demo) · model: mock-1 · 2026-09-29 · 32 test cases

NOT READY

lower bound of accuracy is 61% (below a 70% floor)

Headline accuracy

78%

Task accuracy 95% CI 61–89%
25 of 32 correct

100%

Run-to-run agreement (3 samples/case)

The confidence interval is the honest range. With 32 cases you can distinguish only fairly large differences; ≈1568 cases are needed to reliably detect a 5-point gap between two versions.

Accuracy by category

Category	Score	Accuracy (95% CI)
billing	5/8	62% CI 31–86%
bug	7/8	88% CI 53–98%
feature	8/8	100% CI 68–100%
general	5/8	62% CI 31–86%

Robustness — does it survive benign input changes?

Each row re-runs the same cases with a meaning-preserving tweak. Big negative drops mean the feature is brittle.

Perturbation	Accuracy	vs. baseline
lowercase	78%	+0 pts
trailing_space	81%	+3 pts
polite_wrap	88%	+9 pts
shuffle_choices	78%	+0 pts

Calibration — can you trust its confidence?

0.200

Brier score (lower better)

0.130

ECE (0 = perfectly calibrated)

If ECE is high, the model says "90% sure" while being right far less often — dangerous when a human trusts the confidence.