The terms we use, in plain language.

Tasks and environments

Agentic task

A task where an AI model has to act, not just answer: run commands, edit files, call tools or click through an app, over many steps, to reach a goal. Used to train models (as RL environments) and to test them (as benchmarks).

Instruction also: prompt, task description

What the agent is asked to do, written as a user or operator would say it.

Environment also: sandbox, gym, container

Where the agent works: a container with files and tools, a web app, or a set of tool servers. A good environment behaves the same way on every run.

RL environment also: RL gym

An environment plus tasks and verifiers, built so a model can be trained with reinforcement learning: it attempts the task, the verifier scores it, and the score becomes the reward the model learns from.

Seed data also: initial state, fixtures

The starting data inside an environment, such as the users, tickets and records in a database before the agent begins. Tasks have to be true to their seed data: if the instruction names a user, that user must exist.

Tool-use task

An agentic task where the agent works by calling tools (APIs), often against a seeded database: "find the open ticket for this customer and escalate it". Usually scored by checking the database afterwards.

Computer-use task also: UI task, browser task

An agentic task done through a graphical interface, such as a web app, by clicking and typing. Scored by comparing the app's state before and after.

Reference solution also: oracle, golden solution, golden trajectory

A known-good way to complete the task, shipped with it. It proves the task is solvable, and it's the example the verifiers are checked against. "Golden trajectory" is the step-by-step version for tool-use tasks.

Persona

The user the agent is acting for in a tool-use task, such as "an office manager booking travel". The instruction, permissions and data should all fit that persona.

Checking and scoring

Verifier also: tests, grader, checker, reward function

The check that decides whether the agent succeeded, usually by inspecting the final state of the environment. It can be code (unit tests, a SQL query) or a model acting as judge. If the verifier is wrong, every score built on it is wrong.

SQL verifier

A verifier written as a database query. For example, after a "close this ticket" task, it checks that the ticket's status is now "closed".

A common fault: it checks that the ticket exists instead of checking its status, so it passes even when the agent did nothing.

LLM judge also: LLM-as-a-judge, model grader

A language model asked to score an output or decide whether a task was done, guided by a prompt or rubric. Useful where code can't decide, unreliable when used alone with a vague prompt.

Rubric

A written standard for judging quality: the criteria, the scale, the pass threshold, and examples of passing and failing work. Rubrics are used by human reviewers and by LLM judges.

Reward also: score

The number a verifier produces for a run: pass or fail, or partial credit. In RL training, it's the signal the model learns from, so a wrong reward teaches the wrong behaviour.

Empty solution also: nop, null run, do-nothing baseline

A run where the agent does nothing at all. It should always fail. If it passes, the verifiers aren't checking the outcome.

State diff

The difference between an environment's state before and after a run: which files, rows or app settings changed. Used to check that the agent did what was asked, and nothing else.

Answer leakage

When the expected answer or test logic is visible to the agent, for example copied into its container. The agent can then pass without solving the task.

Model runs

Trajectory also: trace, transcript

The step-by-step record of an agent working on a task: its reasoning, each tool call or command, what came back, and its final answer.

The word is used for two different things. A trajectory can be training data (an example written or curated by people, see ReAct), or the record of a model run (a rollout). We say which one we mean.

Rollout also: run, trial, episode, attempt

One attempt by a model at one task, producing a trajectory and a reward.

Job

A batch of rollouts, for example five attempts at each of a hundred tasks. Patterns across a job, such as every attempt failing at the same step, often point to a broken task rather than a weak model.

Reward hacking

When a model gets a passing score without really doing the task, by exploiting a weakness in the verifier or environment. Examples: editing the tests, hard-coding the expected output, or reading a leaked answer.

Failure attribution

Working out whose fault a failed run was: the model's (a genuine mistake), the task's (an unclear instruction or over-strict verifier), or the infrastructure's (a tool server that timed out). Only the first should count against the model.

Near miss

A run that did essentially the right thing but still failed, often because the verifier accepts only one exact answer or format. A sign the verifier is too strict.

Training data

SFT supervised fine-tuning

Training a model on examples of good responses, usually written or edited by people.

RLHF reinforcement learning from human feedback

Training a model using human judgments of its outputs, typically by comparing two responses and picking the better one (preference data).

RL with verifiable rewards

Training a model on tasks where success can be checked automatically by a verifier, such as agentic tasks, code with tests or maths with known answers. The quality of the verifiers sets the quality of the training.

ReAct trajectory also: tool-use trajectory, agentic SFT data

A training example showing a model alternating between reasoning ("thought"), acting (a tool call) and reading the result ("observation") until it answers. Named after the ReAct (reason plus act) pattern.

CoT chain of thought, reasoning trace

Written step-by-step reasoning that leads to an answer. CoT data teaches models to reason, so every step must be valid, not just the final answer.

Annotation

Human work that produces or labels training data: writing responses, rating outputs, ranking pairs, or building tasks.

Benchmarks and tools

Harbor

An open-source framework for running and evaluating agents on containerized tasks. It defines a standard task format (instruction, environment, tests and a reference solution) and runs agents against tasks locally in Docker or in cloud sandboxes. Terminal-Bench is built on it, and many other task sets use its format.

Terminal-Bench

An open-source benchmark of hard tasks that an agent has to complete in a terminal, inside a container, such as building software, fixing systems or analysing data.

SWE-bench

An open-source benchmark built from real GitHub issues. The agent gets a repository and an issue and must produce a code change that fixes it, which is checked by running the project's tests.

Agent skill

A packaged folder of instructions, scripts and resources that an agent can load to handle a specialised job. Skill tasks test whether the agent finds and uses the right one.

MCP Model Context Protocol

An open standard for connecting models to tools and data through "MCP servers". Many tool-use environments expose their tools this way.

Sandbox

An isolated place to run a task safely, such as a local Docker container or a cloud sandbox service.

Benchmark and tool names belong to their respective projects. Rubric Agents is not affiliated with or endorsed by them.

Our review stack

Deterministic check also: coded check, lint

A rule checked by code, with no model involved: a required file exists, a query only reads, a referenced column is real. Fast, cheap, and always gives the same answer.

Dimension

One distinct aspect of quality, judged on its own, such as prompt quality, verifier quality or reasoning. We split a quality standard into dimensions so every failure has a name.

Dimension agent

Our term for an LLM reviewer that scores one dimension only, against its own rubric and threshold, and explains its score. Where useful, it works inside the task's environment so it can inspect files and run commands.

Consensus review

Running the same rubric with several independent model reviewers and combining their verdicts. We use three model reviewers, and a criterion fails only when at least two flag it. Human experts are used separately, to calibrate the rubrics.

Calibration also: agreement

Comparing agent verdicts with expert human judgments on the same items, and tuning rubrics until they agree. It's how you know the automated review can be trusted.

Threshold

The pass line for a dimension. A score below it flags the item.

Routing

The outcome we give every task at the end of review: ready for evaluation, needs a human (with a report on what to fix), or failed because of infrastructure.

Automatic repair

Fixing a fault, currently verifiers, without a person: a model proposes a fix, coded checks accept or reject it, and the task is re-run to prove it.