Independent audits of public benchmarks.
We run our review stack on public agentic benchmarks and publish what we find. Every finding comes with the run that proves it, or the file and line it rests on.
-
Every audited task can be mis-scored
All 13 tasks have a critical or major finding that can change a leaderboard score: 51 critical, 76 major. The tasks need H100 or B200 GPUs and verifiers that run for up to 24 hours, so this was a code review. Each finding cites the file and line it rests on.
Example, analyst-checkedIn libexpat-to-x86asm the reward script maps only one way of registering tests, so 216 of the bundled tests never count toward any module's score.
-
The 30 lowest-pass-rate tasks
Of DeepSWE's 30 hardest tasks, 6 reject correct or reasonable alternative implementations. 13 more have reviewer findings not yet verified by execution.
ExampleThe reference solution with only a rule's display name changed, from "Auto Table of Contents" to "Auto TOC". Verifier reward: 0.
-
Pilot on the lowest-scoring tasks
Four tasks with near-zero reward across frontier models, plus one high-pass-rate control. Four of the five have a critical or major defect. The control came back clean.
ExampleThe reference solution with only one rejection reason swapped for an equally decisive one. 227 of 232 points, scored all-or-nothing. Verifier reward: 0.
-
Pilot with a control task
Four low-pass-rate tasks and one 92% control, to check the reviewer doesn't over-flag. Two tasks have defects, both verified by execution. The control came back clean.
Pass rates describe each sample, not the whole benchmark. Benchmark names belong to their respective projects; we're not affiliated with or endorsed by them.
Every finding can be checked.
- Review. Reviewer agents read the instruction, tests and environment against a rubric.
- Reproduce. We change the reference solution as little as possible, so it is still correct, and run the task's own verifier.
- Report. If a correct solution scores zero, the finding is marked verified by execution. When a benchmark can't be run in our sandbox, findings cite the file and line instead, and say what still needs a run.
Send us 20 tasks. We'll show you which ones are broken, and why.
A free 20-task pilot. Book a 20-minute call to start.