Consider a constructed acquisition case built around a familiar failure pattern: an AI product posts 98% accuracy on a curated evaluation set, then performs at roughly 71% when measured on live customer traffic. Both numbers can be honestly produced. They answer different questions.
The point is not that a particular seller lied or that a particular transaction failed. It is that a model can be impressive inside the conditions used to produce a headline metric and materially weaker under the conditions a buyer actually cares about. A diligence process that verifies the software exists without re-measuring the claim under decision-relevant conditions can miss that distinction.
I spend my time on that gap. What follows is a teardown of a claim — how a represented advantage comes apart when you actually re-run it — worked on a deliberately constructed target so I can show every number. The target is invented. The method is not; the same method runs publicly, today, on research claims, and I’ll show you that at the end.
01Why the number moves
AI-enabled software can command a premium when buyers believe the product has a proprietary model, a data moat, or a measurable performance edge. Traditional technical diligence is often optimized for questions such as whether the software exists, whether the architecture is supportable, and whether the product can be operated securely. A separate measurement question remains: does the claimed performance survive outside the conditions used to produce the headline?
A model’s headline metric is a measurement, and every measurement carries the fingerprints of how it was taken. Change the comparator, the customer mix, or the slice of time, and the number can move. A buyer that cares about transfer and durability therefore needs to re-take the measurement under conditions the seller did not choose.
Reproducing a number and verifying the decision-relevant claim are different tasks.
02Four tests
Independent verification of a technical claim comes down to four questions, asked in order. Each one is a place a represented advantage can leak.
Real. Does it reproduce outside the seller’s environment? Rebuild the claim from the evidence in the room and run it yourself. A result that only holds inside the vendor’s own harness is not yet sufficient evidence for the acquisition thesis.
Proprietary. Is the advantage the target’s architecture, or a handicapped comparator? Normalize the baseline and measure how much of the gap survives.
Durable. Does it survive transfer? Re-measure on fresh customers, shifted distributions, and later time periods. An edge that depends heavily on its design cohort deserves less underwriting weight.
Economic. Narrow or fully loaded? Bridge the unit economics from the represented metric to inference cost, retries, human review, retention, and other costs that determine the margin actually earned.
03The teardown
Here is what those four questions do to a real-looking claim. Call the target Meridian — a constructed vertical AI-SaaS business. Management represents a +18.375-point performance advantage over a baseline, attributed to “proprietary AI and a defensible data moat.” The evidence pack lets us reconstruct the number and take it apart.
Read the two bars against each other and the thesis reorganizes. In the mature cohort, more than half of the represented 18.375-point headline is the gap between a thin management baseline and a normalized generic scaffold: 9.750 points. That portion is a comparator-envelope effect rather than Meridian-specific technology.
The architecture component is real and survives transfer, but it weakens: 5.125 points on mature customers becomes 3.889 points on fresh customers. That is a genuine target-specific contribution. Whether it is durable enough to deserve a premium is a separate competitive-replication question.
The data layer is the most fragile part of the represented moat. Its measured contribution falls from 3.500 points to 0.333 points on fresh customers. In the constructed evidence pack, the data contribution that survives the transfer test is only about 1.8% of the original 18.375-point headline.
None of this says “do not buy Meridian.” It says the underwriting should distinguish a comparator effect, a real-but-weaker architecture effect, and a data contribution that largely fails the fresh-customer transfer test.
04Why this can happen
Sellers are naturally motivated to present strong, supportable measurements. Broader technology-diligence teams have many responsibilities and may not be scoped to reconstruct every quantitative model-performance claim from first principles. The AI premium can therefore depend on a measurement whose comparator, cohort, and transfer properties have not been independently re-tested.
The fix is not necessarily a longer questionnaire. It is a bounded reproduction step: rebuild the claim, normalize the comparison, test transfer, and preserve an evidence trail. In the Meridian example, the arithmetic is inspectable and the conclusion changes because the evidence states are kept separate.
05What a buyer should demand
If you underwrite AI or software and you take one thing from this, take the list. Before an AI premium reaches investment committee, require that someone has:
The five things to re-run before you pay for a moat
- Reproduced the headline number outside the vendor’s harness. Establish the measurement before interpreting it.
- Normalized the comparator. Rebuild the baseline the target measured itself against, configured fairly.
- Tested transfer on fresh customers. Re-measure outside the design cohort and across relevant time or vertical shifts.
- Bridged narrow to fully-loaded economics. Carry the margin story through inference cost, retries, review, fallback, retention, and other decision-relevant costs.
- Left provenance. Every conclusion should trace to a source, artifact, command, or version that another reviewer can inspect.
06The method is public
The same verification discipline runs in the open on research claims.
RCV-Bench scores whether an agent can distinguish reproduced results, deviations, unsupported/fabricated claims, and results that become fragile under a declared perturbation. The benchmark preserves the difference between simple re-execution and actual verification.
The deterministic naive-rerun baseline illustrates the gap:
Reproducing a number is not the same thing as verifying the claim a decision-maker wants to underwrite. That distinction is testable before a price is set.