Every benchmark has broken tasks. We find them.
Rubric Agents is an independent audit lab for RL environments and benchmarks.
A model learns whatever its verifier rewards. When a task rejects a correct solution, or accepts a wrong one, the training run learns the mistake. We read every task, verifier and reference solution before that happens.
What we found
From our public audits, October 2026Each row counts a sample, not the whole benchmark. Open a row for the evidence behind every finding.
Evaluations
All evaluations →SWE-Marathon
All 4 tasks have a defect that can flip the reward.
CADGenBench
2 of 32 inputs fail the benchmark’s own validity check.
FrontierSWE
Every task has a finding that can change its score.
DeepSWE
6 of 30 tasks reject a correct solution.
Terminal-Bench
3 of 5 tasks reject a correct solution.
How we work
A score is only as good as every part behind it. We check them all.
Hover a part to see a real finding from it.
Send us 20 tasks.
We'll tell you which ones are broken, and why. founders@rubricagents.com