Rubric Agents
What we review

We review the whole task, not just the answer.

Tasks and environments, training data and model runs. Every team names them differently, so each one below says what it is and what we catch.

Start here

Every agentic task has the same six parts.

A fault in any one breaks the task, so we review all of them.

  1. Input

    Instruction

    What the agent is asked to do.

    Also: prompt
  2. Setup

    Environment

    Container, web app or tool servers, with starting data.

    Also: sandbox, gym
  3. Run

    Trajectory

    The record of every step the agent takes.

    Also: rollout, trace
  4. Result

    Final state

    Files, rows or app state afterwards.

    Also: end state
  5. Check

    Verifier

    Code or a judge that checks the final state.

    Also: tests, grader
  6. Score

    Reward

    What the model learns from.

    Also: score

A sound task passes its reference solution and fails an empty solution, where the agent does nothing.

Open-source formats we support

  • Harbor
  • Terminal-Bench
  • SWE-bench
  • MCP
  • ReAct
  • CoT

Names belong to their respective projects. We're not affiliated with or endorsed by them.

A. Tasks and environments

Is the task clear, solvable and scored correctly?

Terminal and environment tasks

A containerized task: instruction, environment, tests and a reference solution.

Leak

The expected output ships in the container, so the agent can read it instead of solving the task.

What we check
  • The instruction says what a finished result looks like
  • The environment builds and has everything the task needs
  • Tests match the instruction, and the agent can't see them
  • The task is hard for the right reasons

Also calledHarbor-format, Terminal-Bench-style, containerized tasks

Software engineering tasks

A repository at a fixed commit, an issue to fix, and tests that decide.

Over-specific tests

The test asserts a function name from the reference fix. A correct fix written differently fails.

What we check
  • The issue is solvable without seeing the fix
  • Fail-to-pass tests fail before and pass after
  • Tests check behaviour, not one implementation
  • Pass-to-pass tests are stable, and the reference fix is minimal

Also calledSWE-bench-style, issue-resolution, repo tasks

Skill tasks

Tasks that test whether an agent uses a packaged skill to do work it couldn't do as well without it.

Skill not needed

The instruction spells out every step the skill contains, so skill use is never tested.

What we check
  • The skill is actually needed
  • The skill works in the environment
  • The verifier checks the result the skill makes possible

Tool-use tasks in RL gyms

An agent calls tools or MCP servers against a seeded database. SQL or judge verifiers check the result.

Free pass

A verifier checks that a ticket exists, not that it was closed. It passes before the agent acts.

What we check
  • The prompt fits the environment and persona
  • The golden trajectory uses the right tools sensibly
  • The task matches its starting data
  • Verifiers check the outcome and fail on a do-nothing run

We can also repair these verifiers automatically.

Computer-use and UI tasks

The agent clicks and types in a web app. It's scored on what changed between start and end.

Unverifiable

A rubric rewards "chose the cheapest option", but the final state never records which options were shown.

What we check
  • The prompt is achievable in the app as configured
  • The state diff matches the prompt, and nothing else changed
  • Every rubric point can be confirmed from the final state
B. Training data

Is it correct, complete and up to your guidelines?

Tool-use trajectories

Multi-turn examples where a model reasons, calls tools and answers. Written or curated by people.

Ignored observation

The tool returns two matching customers. The next step acts as if there were one.

What we check
  • The right tools, with valid inputs
  • Each step uses what the last tool returned
  • Every fact in the answer is grounded

Reasoning and CoT data

Step-by-step reasoning to an answer. Every step has to be valid, not just the result.

Right answer, wrong method

Step 3 treats draws as independent when they're without replacement. The number happens to match.

What we check
  • The final answer is correct
  • Every step follows, with nothing skipped
  • The problem is well posed

Responses and preference data

SFT responses and RLHF ratings, scored per dimension with a reason, before a reviewer sees them.

What you get

A score and reason per dimension, feedback to the contributor, and pass rates per project and team.

What we check
  • Accuracy, reasoning and instruction-following
  • Completeness, clarity, formatting and safety
  • For preference data: the chosen response really is better
  • Consistency across contributors and languages
C. Model runs

What do your scores really mean?

Model run analysis

We read every run, pass or fail, to find out why it got its reward.

Not the model's fault

Scored as a model failure, but the tool server timed out on step 4.

Reward hack

The model edits the test file instead of fixing the bug, and passes.

What we check
  • Whose fault a failure was: model, task or infrastructure
  • Reward hacking, near misses and unsupported claims
  • Patterns across a whole batch of runs
Signs you're ready

Sound familiar? Start with an audit.

  • A verifier passed a wrong answer, and you don't know how many others do.
  • Benchmark scores moved, and nobody can say if it was the model or the tasks.
  • Review is your biggest cost or your slowest step.
  • A client found a broken task before you did.

Send us 20 tasks. We'll show you which ones are broken, and why.

A free 20-task pilot. Book a 20-minute call to start.

Book a 20-minute call Email us