One task, five layers of review.
No single check catches everything. Each layer catches what the one before it can't, and every task leaves with a clear outcome.
-
L1
Deterministic checks
Fast, cheap and certain. Run first.
Caught in secondsA verifier already passes on the starting database, before the agent has done anything.
What it checks
- Structure, required files, schema and naming
- Environment config: Dockerfiles, pinned dependencies, timeouts, network
- Placeholders, stray files and answer leakage
- Verifiers are valid and match the seed data
-
L2
Dimension agents
One rubric per quality dimension. Three reviewers, two votes to fail.
3 of 3 reviewers"Verifier 2 checks that the request exists, not that it was closed."
What it checks
- Quality split into dimensions: prompt, golden trajectory, completeness, verifier quality
- Each dimension has its own rubric and pass threshold, built from your guidelines
- Every score comes with a written reason
- A criterion fails only when at least two of three reviewers flag it
-
L3
Execution validation
Nothing passes until it has been run.
Empty run passedAn agent that does nothing scores full marks. The tests only checked that the output file exists.
What it checks
- The reference solution must pass every verifier
- An empty solution, where the agent does nothing, must fail
- A live frontier-model run against the real environment or tool servers
- Local containers or remote sandboxes, optionally across several models
-
L4
Trajectory analysis
Why a run passed or failed, not just whether.
Benchmark faultFour of five runs fail at the same step. The verifier expects a date format the tool never returns.
What it checks
- Whose fault a failure was: the model's, the task's or the infrastructure's
- Reward hacking, near misses and unsupported claims
- A verdict per run, and patterns across the whole batch
-
L5
Repair and route
A clear outcome for every task.
Fixed and re-runVerifier 2 rewritten to check the status change. The empty run now fails. Routed to ready.
What it does
- Safe fixes, starting with broken verifiers, applied automatically
- Every fix checked by code and proven by re-running the task
- Every task routed to ready, needs a human (with a report) or failed (infrastructure)
A single judge score can't tell you what to fix.
| Single LLM judge | Rubric Agents review stack | |
|---|---|---|
| What you get | One overall score | A verdict per layer and dimension, with reasons |
| Runs the task? | No | Yes: reference, empty and live runs |
| Standard | The model's own idea of quality | Your guidelines, as explicit rubrics |
| Trust | You mostly don't | Three reviewers, calibrated against your experts |
Seven agents, one pass line.
The prompt and trajectory are fine, so one overall score would pass this task. The verifier-quality agent catches that it rewards doing nothing.
- Prompt quality 0.90 Pass
- Golden trajectory 0.93 Pass
- Completeness 0.86 Pass
-
Verifier quality
0.42
Fail
3 of 3 model reviewersVerifier 2 checks that the request row exists, not that its state changed to closed. It passes before the agent does anything. Assert state = 'closed' for the request the prompt names.
- Grounding in seed data 0.88 Pass
- Instruction-following 0.95 Pass
- Judge verifier quality 0.84 Pass
What a rubric looks like.
Simplified. Real rubrics run to dozens of criteria, built from your guidelines.
Dimension: Verifier quality
Scale 0 to 1 | Pass at 0.80Question this agent answers
Does each verifier pass if and only if the task was actually completed?
Criteria
- Asserts the outcome the prompt asks for, not a side effect
- Fails on the untouched starting state
- Accepts every valid way of completing the task
Checks the request's state is "closed" and the original is untouched.
Checks only that the request exists. "Passes before the agent acts."
From your guidelines to a calibrated system.
- 01
Define
Split your quality bar into dimensions.
- 02
Write
A rubric per dimension. Code checks for the rest.
- 03
Connect
Runs in the same sandboxes your tasks use.
- 04
Calibrate
Tuned until it agrees with your experts.
- 05
Gate
Every change re-run against the baseline.
Send us 20 tasks. We'll show you which ones are broken, and why.
A free 20-task pilot. Book a 20-minute call to start.