AI Reliability Test Kit
Know whether your AI feature actually works.
A demo passing is not a feature working. The Test Kit measures accuracy with confidence intervals, robustness to rewording and option order, run-to-run consistency and calibration, then gives you a scorecard you can show your team or a client.
Get the Test Kit on Gumroad See a sample scorecard Try the free tool first
Why measure this
python run_eval.py. No API key needed for the demo.What it measures
Accuracy, honestly
Wilson and bootstrap intervals on every score, macro-F1, and an estimate of how many cases you would need to detect a given gap.
Robustness
Paraphrase, case, whitespace and option-order perturbations, reported as the drop against your baseline.
Consistency and calibration
Run-to-run agreement across repeated samples, plus Brier score and expected calibration error.
Pick your edition
Starter
- Methodology guide: how to know your AI actually works
- Test-set template and schema
- Sample scorecard
Full Kit
- Everything in Starter
- Runnable harness (standard-library core)
- Accuracy with Wilson and bootstrap intervals, macro-F1
- Robustness, consistency and calibration
- HTML scorecard with Ship / Caution / Not-ready verdict
Pro
- Everything in Full
- CI regression gate: fail the build when quality drops
- Grader and perturbation cookbook
- Team license, up to 10 seats
One-time purchase through Gumroad. No subscription. Yours to keep and modify for your own projects.
See the output before you buy
The sample scorecard is real output from the kit on a 32-case demo classifier, produced by the offline mock model. It is there to show the format and the way uncertainty is reported, not to claim anything about any real model.
What this kit will not do
- It measures the things listed above. It does not test safety, bias, security or domain correctness beyond the cases you supply.
- The quality of the verdict depends on the quality and size of your test set. The scorecard tells you how many cases you would need to detect a given gap.
- There are no customer testimonials on this page because I have not collected any, and I will not invent them.
Questions
Do I need an API key to try it?
No. The kit ships with an offline mock model, so you can run python run_eval.py and see a complete scorecard immediately. The mock is for learning the workflow; it says nothing about your real model.
Which models does it work with?
OpenAI, Anthropic, and any OpenAI-compatible endpoint, including local servers such as Ollama, vLLM and LM Studio, so your test data can stay on your machine.
Do I need to be a developer?
Basic Python is enough: run one command, edit a CSV and a JSON config. The Starter edition is a guide plus templates and can be read without running anything.
Can I use it commercially?
Yes, for your own projects and products. You may not resell the kit itself. Pro includes a team license for up to 10 seats.
What does the verdict mean, and can I trust it?
The verdict is a transparent rule over the measured numbers (for example, the lower bound of the accuracy interval against a threshold you set). It is a decision aid, not a guarantee. A Ship verdict means your test set did not find a problem, not that none exists.
Is there a refund?
Purchases go through Gumroad, which handles payment and delivery. If something does not work, email me using the address in the footer.
Who made this?
Robert Morong, an independent research engineer. The same habits (intervals on every number, pinned versions, public code) are on show in the free tool and in the ARC-Challenge audit linked on this site.
Start free, upgrade if it helps
Run the free option-order check on your own eval today. If it shows a problem, the kit covers the rest of the picture.