A represented “AI advantage” can be a real, code-produced number and still be insufficient for the decision being made. I reproduce the claim, normalize the comparator, test transfer and robustness, and leave a trail another reviewer can audit.
Reproduction-first. Findings are reported whether they strengthen or weaken the represented claim. This is adversarial QA, not purchased endorsement.
A headline metric can be true in its original harness and materially weaker after comparator normalization, transfer testing, or fully loaded cost normalization. The question is not only “is the number real?” but “what decision does it actually support?”
Each one isolates a different way a represented advantage can lose underwriting value.
Reconstruct the represented measurement from the strongest available source artifacts without silently changing the protocol.
Normalize the comparator and separate base-model, scaffold, architecture, data, and configuration effects where the evidence permits.
Re-measure on fresh customers, shifted distributions, later periods, or declared semantics-preserving perturbations.
Bridge narrow technical metrics to retries, failed attempts, human review, fallback, and other costs that matter to the economic thesis.
The triage is a bounded first read from public evidence and non-confidential material. Deeper work is scoped to the claim, available access, and decision timeline.
A fast read on whether a represented AI or data-moat claim is verifiable and which failure modes deserve testing first.
One material technical claim taken through reproduction, normalization, targeted stress tests, and an auditable findings package.
A bounded verification layer across multiple AI, data, benchmark, or unit-economics claims that materially affect the thesis.
On RCV-Bench, a deterministic policy that simply re-runs the headline result gets 9/14 verdicts right while finding 0/3 fragile cases and 0/2 fabricated cases. That is the gap between re-execution and verification.
ARC-AGI-2, DESI DR2, and MMLU examples are public with frozen results and explicit limits. The FAIR Universe work remains an external-scoring project in progress; its canonical repository is the authority for whether a Codabench result has been publicly recorded.
Send the non-confidential version first. The triage is free and designed to be useful even if no engagement follows.
Start a free claim triage →