Rubric Agents
QA for agentic tasks, RL environments and LLM outputs

Find the broken tasks before your model trains on them.

Every task is checked, run and repaired, then routed to ready, needs a human, or failed.

Book a 20-minute call See our evaluations
freight-dispatch-shift  Terminal-Bench Real finding →
  1. Deterministic checks 38 checks, no findings
    Pass
  2. Rubric review Correct alternative answers are rejected
    Flag
  3. Reference run Shipped solution scores 1
    Pass
  4. Do-nothing run Scores 0, as it should
    Pass
  5. Execution probe An equally valid reason scores 0 (227 of 232 points)
    Proven
Reviewing… Routed: needs a human Issue #2248 filed
Evaluations

We audit public benchmarks. Here's what we found.

Filed upstream: 3 issues on Terminal-Bench #2248 #2249 #2250

Browse all evaluations

Try it

Spot the broken verifier.

Each of these looks fine on a read. Each one scores agents wrongly.

Tool-use task Task 1 of 3

TaskCustomer #4821 wants their unshipped order cancelled and the payment refunded.

def verify(db, run):
    order = db.orders.get(4821)
    assert order.status == "cancelled"
    calls = [c.name for c in run.tool_calls]
    assert calls == ["get_order", "cancel_order", "refund_payment"]
    assert db.refunds.count(order_id=4821) == 1

Line 5 demands one exact sequence of tool calls. An agent that refunds before it cancels ends in the same correct state and still scores zero.

Where do you fit?

Pick your starting point.

The method

Five layers. Every task.

Code checks first. Models judge only what code can't. Nothing passes until it has been run.

  1. L1

    Deterministic checks

    Schema, config, answer leaks, seed data.

  2. L2

    Dimension agents

    One rubric per quality dimension. Three reviewers, two votes to fail.

  3. L3

    Execution

    The reference must pass. A do-nothing run must fail.

  4. L4

    Trajectories

    Why models failed, and whether they gamed the checks.

  5. L5

    Repair and route

    Safe fixes applied. Ready, needs a human, or failed.

How each layer works

"Define the tolerance, measure against it, and don't ship what fails."

Sheharyar Akhtar, founder. Built review infrastructure for frontier AI lab training programs. More about us

Send us 20 tasks. We'll show you which ones are broken, and why.

A free 20-task pilot. Book a 20-minute call to start.

Book a 20-minute call Email us