Judge the work by the evidence.

Agent evaluation · RCV-Bench

Can an AI agent verify a research claim?

A 14-task benchmark separates simple rerunning from harder provenance and robustness judgments. The naive rerun baseline scores 9/14 but misses every FRAGILE and FABRICATED case; a first autonomous local-agent baseline catches one fabricated case but remains weak overall.

The point is diagnostic, not leaderboard theater: can an agent distinguish a reproduced result from a deviation, unsupported evidence, or a result that collapses under a declared robustness test?

Evaluation reliability · Inspect AI

Measure whether a benchmark score is stable

Eval Invariance Engine wraps multiple-choice evaluations with semantics-preserving perturbations and reports score drift and item-level flip rates inside Inspect AI.

It measures a narrow question directly: how much does the evaluation result depend on representation details that are not supposed to change the underlying task?

AI evaluation · MMLU

Option-order robustness

The regenerated Qwen harness changes its underlying answer on roughly 78% of sampled questions under four cyclic option orderings in both bf16 and fp32.

The headline robustness failure regenerated while several historical stability and calibration quantities did not. Tie-breaking and dtype controls did not remove the effect; the unresolved quantities remain explicitly unresolved.

AI evaluation · ARC-Challenge · two model families

Does a model's score depend on option order?

Four small open models (Qwen2.5 0.5B and 1.5B, SmolLM2 360M and 1.7B) were scored on all 1,165 four-option ARC-Challenge test questions under four cyclic option orderings. Within Qwen the order effect shrinks with size (74.6% of answers flip at 0.5B, 31.6% at 1.5B), but SmolLM2-1.7B flips 59.8%, so size alone does not predict it.

Four models are not a scaling law, and precision differs across runs (disclosed, with a tie-break control). SmolLM2-360M scores at chance and is excluded from comparisons. Pinned revisions, raw predictions, and every statistic are generated by code.

Scientific reproducibility · DESI DR2

Stress-test which observation carries the result

De-Stress Lab packages selection-aware influence analysis, empirical null calibration, frozen result ledgers, and a reproducible DESI DR2 application.

In the flagship analysis, LRG2 is load-bearing for the observed preference, but its localization is not additional independent evidence after conditioning on the global fluctuation.

Reasoning evaluation · ARC-AGI-2

When candidate selection helps but generation still fails

Corrected same-holdout analysis puts k=1 demonstration reliability at 32.8%. MDL improves ambiguous-case selection by 11.1 points over random, while the candidate oracle is only 3.7 points above MDL.

The public correction overturned the earlier optimistic calibration story. The frozen solver remains 0/167; the contribution is the measurement and selection analysis, not a competitive ARC solver.

RCV-Bench

Run an outside agent

Evaluate an agent on the 14-task protocol under a declared model, tool, network, budget, and isolation policy.

Eval Invariance Engine

Try it on an existing Inspect task

Report whether the wrapper preserves task shape and scoring semantics, plus any observed drift or item-level flips.

MMLU Robustness Audit

Reproduce the pinned protocol

Run the fixed 300-question protocol using the declared model and dataset revisions and compare the regenerated evidence.